Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: Ambiguous error message for wrong -benchtime value #46043

Open
umi0410 opened this issue May 7, 2021 · 2 comments
Open

testing: Ambiguous error message for wrong -benchtime value #46043

umi0410 opened this issue May 7, 2021 · 2 comments
Labels
GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@umi0410
Copy link

umi0410 commented May 7, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16.4 linux/amd64

I also checked go1.14 and go1.15 by docker image.

Does this issue reproduce with the latest release?

I guess. I tested with go 1.16.4 docker image(which was built 15 hours ago) and the ambiguous error message was reproduced.

What operating system and processor architecture are you using (go env)?

I think go env doesn't matter so I skip this question.

What did you do?

I tried to repeat benchmark for specific times. I saw there is a flag to set the number of repetition ago but didn't remember how to use it exactly so I just tried -benchtime 1 to execute the benchmark only once.

# main_test.go
package main

import "testing"

func BenchmarkDummy(b *testing.B) {
    for i := 0; i < b.N; i++ {    }
}
$ go test -bench=. -benchtime 1
go test -bench=. -benchtime 1
invalid value "1" for flag -test.benchtime: invalid duration
Usage of /tmp/go-build191827203/b001/me.test:
  -test.bench regexp
        run only benchmarks matching regexp
  -test.benchmem
        print memory allocations for benchmarks
  -test.benchtime d
        run each benchmark for duration d (default 1s)
  -test.blockprofile file
        write a goroutine blocking profile to file
  -test.blockprofilerate rate
        set blocking profile rate (see runtime.SetBlockProfileRate) (default 1)
  -test.count n
        run tests and benchmarks n times (default 1)
 ... // I left out other logs.
exit status 2
FAIL    me      0.001s

What did you expect to see?

I think the error message should let users know the iteration count feature of benchtime flag.

invalid value "1" for flag -test.benchtime or -benchtime: invalid duration or iteration count.
benchtime should be the duration (e.g. ms, s) or the iteration count(e.g. 1x, 100x).
Usage of /tmp/go-build191827203/b001/me.test:
...
  -test.benchtime d
        d can be duration or iteration count(default 1s).
        if d is duration, run each benchmark for duration d.
        else if d is iteration count, run each benchmark iterating for d.
...

What did you see instead?

I pasted the log above.

My additional explanation

The error log said "invalid value "1" for flag -test.benchtime: invalid duration". But benchtime flag can be either duration or iteration count. This ambiguity of benchtime flag has also been introduced in #32051 and robpike hadn't been aware of the feature of benchtime flag as iteration count.

The iteration count feature of benchtime flag was introduced in 1.12 but I think the error message didn't reflect on the update.

@umi0410 umi0410 changed the title cmd/go: Ambiguous error message for wrong -benchtime value testing: Ambiguous error message for wrong -benchtime value May 7, 2021
@umi0410
Copy link
Author

umi0410 commented May 7, 2021

You can refer to this codes. (golang/go/src/testing/benchmark.go#L28)

@jayconrod jayconrod added GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done. labels May 7, 2021
@jayconrod jayconrod added this to the Backlog milestone May 7, 2021
@gopherbot
Copy link

Change https://golang.org/cl/317916 mentions this issue: testing: Improve ambiguous error message about wrong value for -benchtime flag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

3 participants