Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: calculate 'grain' in different way if specifying -benchtime=Nx #37996

Open
shawndx opened this issue Mar 22, 2020 · 0 comments

Comments

@shawndx
Copy link
Contributor

shawndx commented Mar 22, 2020

Benchmarks running with 'RunParallel' calculates the 'grain' value based on the previous iteration number and previous duration, while the value might be fuzzy if -benchtime=Nx is specified since the auto adjustment phase is skipped and the previous iteration number is always 1, resulting in a much smaller 'grain' (most of time it's 1) than normal flow.

example 1:

// normal flow
$ go test -run=none -bench=BenchmarkChanNonblocking -count=5 runtime
BenchmarkChanNonblocking-40     1000000000               0.570 ns/op
BenchmarkChanNonblocking-40     1000000000               0.386 ns/op
BenchmarkChanNonblocking-40     1000000000               0.396 ns/op
BenchmarkChanNonblocking-40     1000000000               0.374 ns/op
BenchmarkChanNonblocking-40     1000000000               0.383 ns/op

// running with -benchtime=1000000000x delivers
$ go test -run=none -bench=BenchmarkChanNonblocking -count=5 -benchtime=1000000000x runtime
BenchmarkChanNonblocking-40     1000000000              23.6 ns/op
BenchmarkChanNonblocking-40     1000000000              22.5 ns/op
BenchmarkChanNonblocking-40     1000000000              22.0 ns/op
BenchmarkChanNonblocking-40     1000000000              22.1 ns/op
BenchmarkChanNonblocking-40     1000000000              22.5 ns/op
example 2:
$ go test -run=none -bench=BenchmarkMatchParallelShared -count=5 regexp
BenchmarkMatchParallelShared-40         77046073                14.4 ns/op
....

$ go test -run=none -bench=BenchmarkMatchParallelShared -count=5 -benchtime=77046073x regexp
BenchmarkMatchParallelShared-40         77046073                75.9 ns/op
....

Propose to calculate 'grain' by "b.benchTime.n / numProcs" first if -benchtime=Nx is specified, then continue with the current rounding up.

diff --git a/src/testing/benchmark.go b/src/testing/benchmark.go
index 88ba0f0242..d35764b3b9 100644
--- a/src/testing/benchmark.go
+++ b/src/testing/benchmark.go
@@ -735,8 +735,11 @@ func (b *B) RunParallel(body func(*PB)) {
        // Calculate grain size as number of iterations that take ~100<C2><B5>s.
        // 100<C2><B5>s is enough to amortize the overhead and provide sufficient
        // dynamic load balancing.
+       numProcs := b.parallelism * runtime.GOMAXPROCS(0)
        grain := uint64(0)
-       if b.previousN > 0 && b.previousDuration > 0 {
+       if b.benchTime.n > 0 {
+               grain = uint64(b.benchTime.n / numProcs)
+       } else if b.previousN > 0 && b.previousDuration > 0 {
                grain = 1e5 * uint64(b.previousN) / uint64(b.previousDuration)
        }
        if grain < 1 {
@@ -749,7 +752,6 @@ func (b *B) RunParallel(body func(*PB)) {
        }

        n := uint64(0)
-       numProcs := b.parallelism * runtime.GOMAXPROCS(0)

New output with the above change:

$ go test -run=none -bench=BenchmarkChanNonblocking -count=5 -benchtime=1000000000x runtime
BenchmarkChanNonblocking-40     1000000000               0.553 ns/op
BenchmarkChanNonblocking-40     1000000000               0.373 ns/op
BenchmarkChanNonblocking-40     1000000000               0.373 ns/op
BenchmarkChanNonblocking-40     1000000000               0.371 ns/op
BenchmarkChanNonblocking-40     1000000000               0.372 ns/op

$ go test -run=none -bench=BenchmarkMatchParallelShared -count=5 -benchtime=77046073x regexp
BenchmarkMatchParallelShared-40         77046073                14.8 ns/op
BenchmarkMatchParallelShared-40         77046073                14.3 ns/op
BenchmarkMatchParallelShared-40         77046073                14.0 ns/op
BenchmarkMatchParallelShared-40         77046073                13.8 ns/op
BenchmarkMatchParallelShared-40         77046073                13.8 ns/op
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Incoming
Development

No branches or pull requests

2 participants