Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proposal: benchmarks or benchstat: should question timings less than 1ns #59926

Closed
bboreham opened this issue May 2, 2023 · 8 comments
Closed

proposal: benchmarks or benchstat: should question timings less than 1ns #59926

bboreham opened this issue May 2, 2023 · 8 comments
Labels
Proposal
Milestone
Proposal

Comments

@bboreham
Copy link
Contributor

bboreham commented May 2, 2023

From time to time I see someone quote a benchmark output with a timing like "0.03995ns".

Invariably this means the benchmark is doing nothing at all, e.g. because they forgot to use b.N.

Similarly to when benchstat advises to use more iterations to get a stable result, it could advise that such a short timing may be an error.
@gopherbot gopherbot added this to the Proposal milestone May 2, 2023
@ianlancetaylor
Copy link
Contributor

CC @aclements

@rsc
Copy link
Contributor

rsc commented Jun 21, 2023

I had an issue somewhere, which I can't find right now, to report when a benchmark seems to be ignoring b.N. Diagnosing that in package testing seems like it makes more sense than doing it in benchstat.

@ianlancetaylor
Copy link
Contributor

Perhaps #38677 ?

@rsc
Copy link
Contributor

rsc commented Jun 21, 2023

CL 230978 is what I was thinking of. I think it would catch anything that would produce 0.03995ns.

@rsc
Copy link
Contributor

rsc commented Jun 21, 2023

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc
Copy link
Contributor

rsc commented Jun 28, 2023

Austin pointed out various problems with my CL so I wrote a vet check instead. This is now probably a duplicate of #38677.

@bboreham
Copy link
Contributor Author

One case I don’t think would be covered is when the compiler optimizes away your entire loop.

@rsc
Copy link
Contributor

rsc commented Jun 28, 2023

This proposal is a duplicate of a previously discussed proposal, as noted above,
and there is no significant new information to justify reopening the discussion.
The issue has therefore been declined as a duplicate.
— rsc for the proposal review group

@rsc rsc closed this as completed Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Proposal
Projects
Proposals
Status: Declined
Milestone
Proposal
Development

No branches or pull requests
4 participants
@rsc @ianlancetaylor @bboreham @gopherbot