Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/perf/cmd/benchstat: spurious "missing iteration count" warning #58528

Closed
elagergren-spideroak opened this issue Feb 14, 2023 · 6 comments
Closed
Labels
NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@elagergren-spideroak
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.20 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

n/a

What did you do?

Given this program https://go.dev/play/p/8rhbHTxDF4C

$ go test -v -vet all -failfast -bench . -count=10 2>&1 | tee old.txt
[...]
$ go test -v -vet all -failfast -bench . -count=10 2>&1 | tee new.txt
$ benchstat old.txt new.txt
old.txt:5: missing iteration count
old.txt:16: missing iteration count
new.txt:5: missing iteration count
new.txt:16: missing iteration count
goos: darwin
goarch: amd64
pkg: example.com/issue
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
        │   old.txt    │               new.txt               │
        │    sec/op    │   sec/op     vs base                │
A-12      25.735m ± 0%   1.137m ± 3%  -95.58% (p=0.000 n=10)
B-12      25.704m ± 0%   1.131m ± 1%  -95.60% (p=0.000 n=10)
geomean    25.72m        1.134m       -95.59%

What did you expect to see?

Just the benchmark.

What did you see instead?

old.txt:5: missing iteration count
old.txt:16: missing iteration count
new.txt:5: missing iteration count
new.txt:16: missing iteration count
@gopherbot gopherbot added this to the Unreleased milestone Feb 14, 2023
@prattmic
Copy link
Member

The file look like:

goos: linux
goarch: amd64
pkg: example.com/nop/issue
cpu: Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz
BenchmarkA
BenchmarkA-12               1074           1114089 ns/op
BenchmarkA-12               1059           1116605 ns/op
BenchmarkA-12               1069           1116068 ns/op
BenchmarkA-12               1082           1109577 ns/op
BenchmarkA-12               1092           1114838 ns/op
BenchmarkA-12               1057           1127497 ns/op
BenchmarkA-12               1068           1112353 ns/op
BenchmarkA-12               1082           1119847 ns/op
BenchmarkA-12               1069           1109970 ns/op
BenchmarkA-12               1076           1119566 ns/op
BenchmarkB
BenchmarkB-12               1086           1115953 ns/op
BenchmarkB-12               1054           1115512 ns/op
BenchmarkB-12               1120           1111781 ns/op
BenchmarkB-12               1080           1122841 ns/op
BenchmarkB-12               1081           1110178 ns/op
BenchmarkB-12               1069           1116808 ns/op
BenchmarkB-12               1068           1110374 ns/op
BenchmarkB-12               1072           1119656 ns/op
BenchmarkB-12               1058           1102025 ns/op
BenchmarkB-12               1108           1082698 ns/op
PASS
ok      example.com/nop/issue   26.259s

The problem is the header BenchmarkA and BenchmarkB lines.

cc @aclements

@prattmic prattmic added the NeedsFix The path to resolution is known, but the work has not been done. label Feb 14, 2023
@aclements
Copy link
Member

Well it's not wrong that those lines are missing the iteration count. I guess it could see that there are also no measurements and ignore the lines. But where are the lines coming from? Is it the -v passed to go test?

@elagergren-spideroak
Copy link
Author

@aclements yep

@elagergren-spideroak
Copy link
Author

@aclements iirc this didn't occur with the previous benchstat, so it's an (admittedly minor) regression.

@maggie44
Copy link

It seems for me the -bench is turning on the verbose mode, which is resulting in my code logging to the console which then gets pushed in between the expected output and creates issues with the expected formatting:


The second, called package list mode, occurs when go test is invoked
with explicit package arguments (for example 'go test math', 'go
test ./...', and even 'go test .'). In this mode, go test compiles
and tests each of the packages listed on the command line. If a
package test passes, go test prints only the final 'ok' summary
line. If a package test fails, go test prints the full test output.
If invoked with the -bench or -v flag, go test prints the full
output even for passing package tests, in order to display the
requested benchmark results or verbose logging. After the package
tests for all of the listed packages finish, and their output is
printed, go test prints a final 'FAIL' status if any package test
has failed.

@gopherbot
Copy link

Change https://go.dev/cl/540795 mentions this issue: benchfmt: ignore just benchmark name on a line

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

5 participants