Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/perf/benchstat: no group headers shown if only one group #33169

Closed
shawndx opened this issue Jul 18, 2019 · 3 comments
Closed

x/perf/benchstat: no group headers shown if only one group #33169

shawndx opened this issue Jul 18, 2019 · 3 comments
Labels
FeatureRequest FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@shawndx
Copy link
Contributor

shawndx commented Jul 18, 2019

What version of Go are you using (go version)?

$ go version
go version devel +a6a7b148f8 Mon Jul 15 23:00:52 2019 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/xiaji01/.cache/go-build"
GOENV="/home/xiaji01/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/xiaji01/.go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/mnt/share/homes/xiaji01/go1.12/x86"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/mnt/share/homes/xiaji01/go1.12/x86/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build378631038=/tmp/go-build -gno-record-gcc-switches"

What did you do?

#install benchstat
go get -d golang.org/x/perf/cmd/benchstat
go install golang.org/x/perf/cmd/benchstat

#run benchmark of a SINGLE package under /src
go test -count 5 -timeout 20m -run=^$ -bench=. crypto/sha256 | tee bench.log

#run benchstat to do analysis, using default options
benchstat bench.log

#output of the benchstat doesn't have group header (pkg, goos, goarch)
name time/op
Hash8Bytes-40 294ns ± 3%
Hash1K-40 3.50µs ± 5%
Hash8K-40 26.2µs ± 4%

name speed
Hash8Bytes-40 27.2MB/s ± 3%
Hash1K-40 293MB/s ± 4%
Hash8K-40 313MB/s ± 4%

What did you expect to see?

Expect benchstat to output group header even the benchmarking data only contains one group.
Since there are a few Go benchmarks from different packages sharing the same name, for example, both crypto/sha256 and crypto/sha512 have benchmarks "BenchmarkHash8Bytes", "BenchmarkHash1K", etc., having a package name, which is part of the group header by default, would facilitate recording their performance data in a database.

We can fulfill the above requirement via multiple ways, just wondering if having benchstat support it is a better option.

The restriction was introduced in commit add18dd,

  •                           if len(c.Groups) > 1 {
    
  •                                   // Show group headers if there is more than one group.
    
  •                                   row.Group = key.Group
    

Any comment is highly appreciated.

Attaching two benchmark log files for your reference.

bench_one_pkg.log: benchmark log of package crypto/sha256, benchstat's output has no group

bench_two_pkgs.log: benchmark log of packages crypto/sha256 and crypto/sha512, benchstat's output has groups.
bench_one_pkg.log
bench_two_pkgs.log

What did you see instead?

@gopherbot gopherbot added this to the Unreleased milestone Jul 18, 2019
@katiehockman
Copy link
Contributor

/cc @rsc

@katiehockman katiehockman added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 29, 2019
@katiehockman
Copy link
Contributor

/cc @aclements

@gopherbot
Copy link

Change https://golang.org/cl/309969 mentions this issue: cmd/benchstat: new version of benchstat

zchee pushed a commit to zchee/golang-perf that referenced this issue Nov 28, 2021
This is a complete rewrite of benchstat. Basic usage remains the same,
as does the core idea of showing statistical benchmark summaries and
A/B comparisons in a table, but there are several major improvements.

The statistics is now more robust. Previously, benchstat used
IQR-based outlier rejection, showed the mean of the reduced sample,
its range, and did a non-parametric difference-of-distribution test on
reduced samples. Any form of outlier rejection must start with
distributional assumptions, in this case assuming normality, which is
generally not sound for benchmark data. Hence, now benchstat does not
do any outlier rejection. As a result, it must use robust summary
statistics as well, so benchstat now uses the median and confidence
interval of the median as summary statistics. Benchstat continues to
use the same Mann-Whitney U-test for the delta, but now does it on the
full samples since the U-test is already non-parametric, hence
increasing the power of this test.

As part of these statistical improvements, benchstat now detects and
warns about several common mistakes, such as having too few samples
for meaningful statistical results, or having incomparable geomeans.

The output format is more consistent. Previously, benchstat
transformed units like "ns/op" into a metric like "time/op", which it
used as a column header; and a numerator like "sec", which it used to
label each measurement. This was easy enough for the standard units
used by the testing framework, but was basically impossible to
generalize to custom units. Now, benchstat does unit scaling, but
otherwise leaves units alone. The full (scaled) unit is used as a
column header and each measurement is simply a scaled value shown with
an SI prefix. This also means that the text and CSV formats can be
much more similar while still allowing the CSV format to be usefully
machine-readable.

Benchstat will also now do A/B comparisons even if there are more than
two inputs. It shows a comparison to the base in the second and all
subsequent columns. This approach is consistent for any number of
inputs.

Benchstat now supports the full Go benchmark format, including
sophisticated control over exactly how it structures the results into
rows, columns, and tables. This makes it easy to do meaningful
comparisons across benchmark data that isn't simply structured into
two input files, and gives significantly more control over how results
are sorted. The default behavior is still to turn each input file into
a column and each benchmark into a row.

Fixes golang/go#19565 by showing all results, even if the benchmark
sets don't match across columns, and warning when geomean sets are
incompatible.

Fixes golang/go#19634 by no longer doing outlier rejection and clearly
reporting when there are not enough samples to do a meaningful
difference test.

Updates golang/go#23471 by providing more through command
documentation. I'm not sure it quite fixes this issue, but it's much
better than it was.

Fixes golang/go#30368 because benchstat now supports filter
expressions, which can also filter down units.

Fixes golang/go#33169 because benchstat now always shows file
configuration labels.

Updates golang/go#43744 by integrating unit metadata to control
statistical assumptions into the main tool that implements those
assumptions.

Fixes golang/go#48380 by introducing a way to override labels from the
command line rather than always using file names.

Change-Id: Ie2c5a12024e84b4918e483df2223eb1f10413a4f
@golang golang locked and limited conversation to collaborators Jan 13, 2024
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FeatureRequest FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

3 participants