Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/perf/cmd/benchstat: wrong title shown for benchstat when comparing memory #58025

Closed
eltonsv opened this issue Jan 26, 2023 · 5 comments
Closed

x/perf/cmd/benchstat: wrong title shown for benchstat when comparing memory #58025

eltonsv opened this issue Jan 26, 2023 · 5 comments
Labels
FrozenDueToAge Tools This label describes issues relating to any tools in the x/tools repository. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Unreleased

Comments

@eltonsv
Copy link

eltonsv commented Jan 26, 2023 •
edited

The output of benchstat does not show the title B/op and instead it shows allocs/op.

What version of Go are you using (go version)?

$ go version
go version go1.19.2 darwin/amd64

Does this issue reproduce with the latest release?

Havent checked yet.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/e.savio/Library/Caches/go-build"
GOENV="/Users/e.savio/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/e.savio/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/e.savio/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.19.2/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.19.2/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.19.2"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="...go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/44/79ttc6d962x84fw3_2g0wd2sdh29lk/T/go-build4279869546=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

When benchmarking a program

$ go test -run='^$' -bench=. -benchmem -count=10 > old.txt

After making changes

$ go test -run='^$' -bench=. -benchmem -count=10 > new.txt

What did you expect to see?

$ ~/go/bin/benchstat old.txt new.txt                                           
name                old time/op    new time/op    delta                                                                           
{entries}                                                                                                                                  

name                old B/op   new B/op   delta                                                                           
{entries}                                                                                                                                  

name                old allocs/op  new allocs/op  delta                                                                           
{entries}

What did you see instead?

$ ~/go/bin/benchstat old.txt new.txt                                           
name                old time/op    new time/op    delta                                                                           
{entries}                                                                                                                                  

name                old allocs/op   new allocs/op   delta                                                                           
{entries}                                                                                                                                  

name                old allocs/op  new allocs/op  delta                                                                           
{entries}
@AlexanderYastrebov
Copy link
Contributor

https://pkg.go.dev/golang.org/x/tools/cmd/benchcmp

Deprecated: benchcmp is deprecated in favor of benchstat: golang.org/x/perf/cmd/benchstat

Recent version of benchstat seems to show units correctly (see also golang/perf@02c5517)

@seankhliao seankhliao changed the title tools/benchcmp: Wrong title shown for benchcmp when comparing memory x/tools/cmd/benchcmp: wrong title shown for benchcmp when comparing memory Jan 28, 2023
@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Jan 28, 2023
@gopherbot gopherbot added this to the Unreleased milestone Jan 28, 2023
@seankhliao
Copy link
Member

which version of benchcmp are you using?

@seankhliao seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jan 28, 2023
@eltonsv
Copy link
Author

eltonsv commented Jan 28, 2023 •
edited

Oops, sorry. I mistitled it as benchcmp instead of benchstat which I actually use for comparing benchmarks.
Will update it as benchstat in the issue title.

@eltonsv eltonsv changed the title x/tools/cmd/benchcmp: wrong title shown for benchcmp when comparing memory x/perf/cmd/benchstat: wrong title shown for benchstat when comparing memory Jan 28, 2023
@seankhliao
Copy link
Member

Which version of benchstat?

Example output from tip:

goos: linux
goarch: amd64
pkg: crypto/rsa
cpu: 12th Gen Intel(R) Core(TM) i7-1260P
                        │   old.txt    │               new.txt                │
                        │    sec/op    │    sec/op     vs base                │
DecryptPKCS1v15/2048-16   976.3µ ± ∞ ¹   976.3µ ± ∞ ¹       ~ (p=1.000 n=1) ²
DecryptPKCS1v15/3072-16   2.871m ± ∞ ¹   2.871m ± ∞ ¹       ~ (p=1.000 n=1) ²
DecryptPKCS1v15/4096-16   6.383m ± ∞ ¹   6.383m ± ∞ ¹       ~ (p=1.000 n=1) ²
EncryptPKCS1v15/2048-16   122.8µ ± ∞ ¹   122.8µ ± ∞ ¹       ~ (p=1.000 n=1) ²
DecryptOAEP/2048-16       991.5µ ± ∞ ¹   991.5µ ± ∞ ¹       ~ (p=1.000 n=1) ²
EncryptOAEP/2048-16       123.9µ ± ∞ ¹   123.9µ ± ∞ ¹       ~ (p=1.000 n=1) ²
SignPKCS1v15/2048-16      1.010m ± ∞ ¹   1.010m ± ∞ ¹       ~ (p=1.000 n=1) ²
VerifyPKCS1v15/2048-16    125.1µ ± ∞ ¹   125.1µ ± ∞ ¹       ~ (p=1.000 n=1) ²
SignPSS/2048-16           1.019m ± ∞ ¹   1.019m ± ∞ ¹       ~ (p=1.000 n=1) ²
VerifyPSS/2048-16         125.0µ ± ∞ ¹   125.0µ ± ∞ ¹       ~ (p=1.000 n=1) ²
geomean                   580.4µ         580.4µ        +0.00%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal

                        │    old.txt    │                new.txt                │
                        │     B/op      │     B/op       vs base                │
DecryptPKCS1v15/2048-16   14.69Ki ± ∞ ¹   14.69Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
DecryptPKCS1v15/3072-16   17.44Ki ± ∞ ¹   17.44Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
DecryptPKCS1v15/4096-16   18.69Ki ± ∞ ¹   18.69Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
EncryptPKCS1v15/2048-16   7.332Ki ± ∞ ¹   7.332Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
DecryptOAEP/2048-16       14.91Ki ± ∞ ¹   14.91Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
EncryptOAEP/2048-16       7.558Ki ± ∞ ¹   7.558Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
SignPKCS1v15/2048-16      21.12Ki ± ∞ ¹   21.12Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
VerifyPKCS1v15/2048-16    7.078Ki ± ∞ ¹   7.078Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
SignPSS/2048-16           21.52Ki ± ∞ ¹   21.52Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
VerifyPSS/2048-16         7.281Ki ± ∞ ¹   7.281Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
geomean                   12.49Ki         12.49Ki        +0.00%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal

                        │   old.txt   │               new.txt               │
                        │  allocs/op  │  allocs/op   vs base                │
DecryptPKCS1v15/2048-16   54.00 ± ∞ ¹   54.00 ± ∞ ¹       ~ (p=1.000 n=1) ²
DecryptPKCS1v15/3072-16   60.00 ± ∞ ¹   60.00 ± ∞ ¹       ~ (p=1.000 n=1) ²
DecryptPKCS1v15/4096-16   60.00 ± ∞ ¹   60.00 ± ∞ ¹       ~ (p=1.000 n=1) ²
EncryptPKCS1v15/2048-16   29.00 ± ∞ ¹   29.00 ± ∞ ¹       ~ (p=1.000 n=1) ²
DecryptOAEP/2048-16       60.00 ± ∞ ¹   60.00 ± ∞ ¹       ~ (p=1.000 n=1) ²
EncryptOAEP/2048-16       35.00 ± ∞ ¹   35.00 ± ∞ ¹       ~ (p=1.000 n=1) ²
SignPKCS1v15/2048-16      77.00 ± ∞ ¹   77.00 ± ∞ ¹       ~ (p=1.000 n=1) ²
VerifyPKCS1v15/2048-16    28.00 ± ∞ ¹   28.00 ± ∞ ¹       ~ (p=1.000 n=1) ²
SignPSS/2048-16           82.00 ± ∞ ¹   82.00 ± ∞ ¹       ~ (p=1.000 n=1) ²
VerifyPSS/2048-16         33.00 ± ∞ ¹   33.00 ± ∞ ¹       ~ (p=1.000 n=1) ²
geomean                   48.30         48.30        +0.00%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal

@eltonsv
Copy link
Author

eltonsv commented Jan 28, 2023 •
edited

Thanks for the info @AlexanderYastrebov & @seankhliao. I updated benchstat to the latest version and the titles are shown correctly now.

Also @seankhliao, I tried a benchstat -v & benchstat -h and it looks like it does not print out the version number. But nevertheless, my issue is now resolved.

For the same benchmarks it displays the right titles

                   │   old.txt    │               new.txt                │
                   │    sec/op    │    sec/op     vs base                │
{values}...
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                   │    old.txt    │                new.txt                │
                   │     B/op      │     B/op       vs base                │
{values}...
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                   │   old.txt    │               new.txt                │
                   │  allocs/op   │  allocs/op    vs base                │
{values}...
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ need >= 4 samples to detect a difference at alpha level 0.05

Thanks a lot for the help, I would be closing the issue now.

@eltonsv eltonsv closed this as completed Jan 28, 2023
@golang golang locked and limited conversation to collaborators Jan 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge Tools This label describes issues relating to any tools in the x/tools repository. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@AlexanderYastrebov @gopherbot @seankhliao @eltonsv