Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmarks run twice with -benchtime 1x #42952

Closed
seebs opened this issue Dec 2, 2020 · 2 comments
Closed

benchmarks run twice with -benchtime 1x #42952

seebs opened this issue Dec 2, 2020 · 2 comments

Comments

@seebs
Copy link
Contributor

seebs commented Dec 2, 2020

reproduced with go1.15, but I suspect it's always been that way since -benchtime was introduced.

Sample code:
https://play.golang.org/p/It6iFOHZldA

So, I wouldn't have noticed this, but I have a benchmark for a pathological case, and the thing about the pathological case is it only shows up with reasonably large inputs -- but shows up by exploding runtime into minutes. So, of course, I run with -benchtime 1x.

... And it runs twice.

Looking at the "got here" output, it's pretty obvious why:

N:1, previousN:0, previousDuration:0
=>
N:1, previousN:1, previousDuration:85948

The thing is, when I've explicitly specified a number of times, I probably don't actually need the initial calibrating run.

There's a reasonable workaround (make it a "Test" function which does its own timing), but it'd be really handy to have all those convenient benchmark things, like reporting memory usage. I thought of trying to use testing.Benchmark(), but it does the same thing -- it runs the function twice.

The best alternative I can think of is to make the exploding parameter a b.N value, and run with -benchtime 1000000x or something like that, but this is completely incompatible with the way any other test/benchmark would want to run, or all the very thoughtful guidance not to vary quadratically on b.N.

@ALTree
Copy link
Member

ALTree commented Dec 2, 2020

I believe this is #32051.

@mvdan
Copy link
Member

mvdan commented Dec 3, 2020

I agree with Alberto, this is a dup, and that issue has plenty of discussion already.

@mvdan mvdan closed this as completed Dec 3, 2020
@golang golang locked and limited conversation to collaborators Dec 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants