Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testing: benchmark verbose output changed in 1.14 #37643

Closed
robaho opened this issue Mar 4, 2020 · 10 comments
Closed

testing: benchmark verbose output changed in 1.14 #37643

robaho opened this issue Mar 4, 2020 · 10 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Go1.14.1

Comments

@robaho
Copy link

robaho commented Mar 4, 2020

Using github.com/robaho/fixed and go bench:

In 1.9.7 results are similar to:

goos: darwin
goarch: amd64
pkg: github.com/robaho/fixed
BenchmarkAddFixed-8         	2000000000	         1.26 ns/op
BenchmarkAddDecimal-8       	 5000000	       322 ns/op
BenchmarkAddBigInt-8        	100000000	        14.5 ns/op
BenchmarkAddBigFloat-8      	20000000	        76.5 ns/op
BenchmarkMulFixed-8         	300000000	         4.24 ns/op
BenchmarkMulDecimal-8       	20000000	        69.5 ns/op
BenchmarkMulBigInt-8        	100000000	        17.0 ns/op
BenchmarkMulBigFloat-8      	50000000	        34.4 ns/op
BenchmarkDivFixed-8         	300000000	         4.59 ns/op
BenchmarkDivDecimal-8       	 2000000	       854 ns/op
BenchmarkDivBigInt-8        	30000000	        54.7 ns/op
BenchmarkDivBigFloat-8      	20000000	       118 ns/op

and in 1.14

goos: darwin
goarch: amd64
pkg: github.com/robaho/fixed
BenchmarkAddFixed
BenchmarkAddFixed-8         	1000000000	         0.519 ns/op
BenchmarkAddDecimal
BenchmarkAddDecimal-8       	 5005356	       236 ns/op
BenchmarkAddBigInt
BenchmarkAddBigInt-8        	82658592	        14.9 ns/op
BenchmarkAddBigFloat
BenchmarkAddBigFloat-8      	15501452	        77.2 ns/op
BenchmarkMulFixed
BenchmarkMulFixed-8         	293514025	         4.02 ns/op
BenchmarkMulDecimal
BenchmarkMulDecimal-8       	16031581	        72.6 ns/op
BenchmarkMulBigInt
BenchmarkMulBigInt-8        	70126652	        17.7 ns/op
BenchmarkMulBigFloat
BenchmarkMulBigFloat-8      	33527037	        35.1 ns/op
BenchmarkDivFixed
BenchmarkDivFixed-8         	265002393	         4.53 ns/op

Notice the extra rows without the PROCs count.
@robaho robaho changed the title go bench output format changed in 14.1 cmd/go: go bench output format changed in 14.1 Mar 4, 2020
@robaho robaho changed the title cmd/go: go bench output format changed in 14.1 cmd/go: go bench output format changed in 1.14 Mar 4, 2020
@bcmills
Copy link
Contributor

bcmills commented Mar 4, 2020

Please fill out the complete issue template. What was the exact command? What does go env report?

@bcmills bcmills added wayit WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed wayit labels Mar 4, 2020
@robaho
Copy link
Author

robaho commented Mar 4, 2020

Sorry, I thought it was a trivial report. I run the tests via Intellij so I didn't change any parameters - just the GOROOT for the project.

In 1.9.7 the command is

GOROOT=/usr/local/Cellar/go@1.9/1.9.7/libexec #gosetup
GOPATH=/Users/robertengels/go #gosetup
/usr/local/Cellar/go@1.9/1.9.7/libexec/bin/go test -c -o /private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go github.com/robaho/fixed #gosetup
/private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go -test.v -test.bench "^BenchmarkAddFixed|BenchmarkAddDecimal|BenchmarkAddBigInt|BenchmarkAddBigFloat|BenchmarkMulFixed|BenchmarkMulDecimal|BenchmarkMulBigInt|BenchmarkMulBigFloat|BenchmarkDivFixed|BenchmarkDivDecimal|BenchmarkDivBigInt|BenchmarkDivBigFloat|BenchmarkCmpFixed|BenchmarkCmpDecimal|BenchmarkCmpBigInt|BenchmarkCmpBigFloat|BenchmarkStringFixed|BenchmarkStringNFixed|BenchmarkStringDecimal|BenchmarkStringBigInt|BenchmarkStringBigFloat|BenchmarkWriteTo$" -test.run ^$ #gosetup

in 1.14 the command is

GOROOT=/usr/local/Cellar/go/1.14/libexec #gosetup
GOPATH=/Users/robertengels/go #gosetup
/usr/local/Cellar/go/1.14/libexec/bin/go test -c -o /private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go github.com/robaho/fixed #gosetup
/private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go -test.v -test.bench "^BenchmarkAddFixed|BenchmarkAddDecimal|BenchmarkAddBigInt|BenchmarkAddBigFloat|BenchmarkMulFixed|BenchmarkMulDecimal|BenchmarkMulBigInt|BenchmarkMulBigFloat|BenchmarkDivFixed|BenchmarkDivDecimal|BenchmarkDivBigInt|BenchmarkDivBigFloat|BenchmarkCmpFixed|BenchmarkCmpDecimal|BenchmarkCmpBigInt|BenchmarkCmpBigFloat|BenchmarkStringFixed|BenchmarkStringNFixed|BenchmarkStringDecimal|BenchmarkStringBigInt|BenchmarkStringBigFloat|BenchmarkWriteTo$" -test.run ^$ #gosetup

@robaho
Copy link
Author

robaho commented Mar 4, 2020

The output of 'go env' is:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/robertengels/Library/Caches/go-build"
GOENV="/Users/robertengels/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/robertengels/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/go-build436672849=/tmp/go-build -gno-record-gcc-switches -fno-common"

@robaho
Copy link
Author

robaho commented Mar 4, 2020

Interesting too that the 'operations' counts are not whole numbers in 1.14

@mengzhuo
Copy link
Contributor

mengzhuo commented Mar 4, 2020

You have to remove -v

#24929
a813d3c7

@bcmills bcmills changed the title cmd/go: go bench output format changed in 1.14 testing: benchmark output format changed in 1.14 Mar 4, 2020
@bcmills
Copy link
Contributor

bcmills commented Mar 4, 2020

@robaho, it seems like the extra lines here are intended to aid in debugging if one of the benchmarks unexpectedly stalls. Other than being different, are they causing any particular harm?

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Mar 4, 2020
@bcmills bcmills changed the title testing: benchmark output format changed in 1.14 testing: benchmark verbose output changed in 1.14 Mar 4, 2020
@robaho
Copy link
Author

robaho commented Mar 4, 2020

@mengzhuo I can't - the parameters passed are part of the IntelliJ GoLand/Plugin test runner.

@bcmills not really, but I noticed strange behavior by the test runner in IntelliJ - the progress lines are blank. I have to assume that other CI tools may have problems due to the format change, additionally the non-whole numbers might cause 'regression alerts' in some CI tools.

Just giving a heads-up - not blocking me in any way.

@bcmills
Copy link
Contributor

bcmills commented Mar 4, 2020

Thanks for letting us know. Doesn't seem like something we should fix in the Go toolchain at this time.

(FYI @dlsniper; this may be something for you to look into on the IntelliJ side.)

@bcmills bcmills closed this as completed Mar 4, 2020
@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 4, 2020
@bcmills bcmills added this to the Go1.14.1 milestone Mar 4, 2020
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 4, 2020
@dlsniper
Copy link
Contributor

dlsniper commented Mar 5, 2020

but I noticed strange behavior by the test runner in IntelliJ - the progress lines are blank.

@robaho I'd love to have more information on this, but I don't think this is the right place. Please file an issue on our tracker at https://youtrack.jetbrains.com/issues/Go or reach out to us via Gophers Slack in the #goland channel (or me personally using the same handle on Gophers Slack or Twitter).

At the moment I'd rather not change anything because I do like the idea of being able to debug stalling benchmarks, which is the reason this was added in the first place.

As for running without -test.v, we can't really do that as it helps us build the UI around the output from go test.

@robaho
Copy link
Author

robaho commented Mar 7, 2020 via email

@golang golang locked and limited conversation to collaborators Mar 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Go1.14.1
Development

No branches or pull requests
5 participants
@dlsniper @mengzhuo @robaho @bcmills @gopherbot