Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testing: benchmarks timing issue when using -count > 1 #25622

Closed
ALTree opened this issue May 29, 2018 · 6 comments
Closed

testing: benchmarks timing issue when using -count > 1 #25622

ALTree opened this issue May 29, 2018 · 6 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Milestone
Go1.11

Comments

@ALTree
Copy link
Member

ALTree commented May 29, 2018 •
edited

@laboger reported in #25520 that 05ca340 introduced timing issues in the results of some benchmarks.

For example, the ChanNonblocking benchmark in runtime is much slower after the 1st iteration on ppc64le

BenchmarkChanNonblocking-16 1000000000 3.15 ns/op
BenchmarkChanNonblocking-16 1000000000 32.7 ns/op

The benchmark uses RunParallel. It seems the the change linked above may have introduced timing issues on benchmarks that use RunParallel. For example, using this:

package p_test

import (
	"bytes"
	"testing"
	"text/template"
)

func BenchmarkF(b *testing.B) {
	templ := template.Must(template.New("test").Parse("Hello, {{.}}!"))
	b.RunParallel(func(pb *testing.PB) {
		var buf bytes.Buffer
		for pb.Next() {
			buf.Reset()
			templ.Execute(&buf, "World")
		}
	})
}

I see on go1.10:

BenchmarkF-4   	10000000	       188 ns/op
BenchmarkF-4   	10000000	       190 ns/op
BenchmarkF-4   	10000000	       189 ns/op
BenchmarkF-4   	10000000	       188 ns/op
BenchmarkF-4   	10000000	       189 ns/op

tip:

BenchmarkF-4   	10000000	       187 ns/op
BenchmarkF-4   	10000000	       198 ns/op
BenchmarkF-4   	10000000	       195 ns/op
BenchmarkF-4   	10000000	       195 ns/op
BenchmarkF-4   	10000000	       194 ns/op

notice how on tip every run after the 1st consistently reports an higher ns/op.
@ALTree ALTree added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels May 29, 2018
@ALTree ALTree added this to the Go1.11 milestone May 29, 2018
@ALTree ALTree mentioned this issue May 29, 2018
Open
@ianlancetaylor ianlancetaylor changed the title test: benchmarks timing issue when using -count > 1 testing: benchmarks timing issue when using -count > 1 May 29, 2018
@laboger laboger mentioned this issue May 30, 2018
Closed
@erifan
Copy link

erifan commented May 31, 2018 •
edited

The same issue on arm64:
goos: linux
goarch: arm64
pkg: encoding/json
BenchmarkTypeFieldsCache/HitTypes1-32 500000000 3.51 ns/op
BenchmarkTypeFieldsCache/HitTypes1-32 500000000 77.9 ns/op
BenchmarkTypeFieldsCache/HitTypes1-32 500000000 64.9 ns/op
BenchmarkTypeFieldsCache/HitTypes10-32 500000000 3.55 ns/op
BenchmarkTypeFieldsCache/HitTypes10-32 500000000 64.6 ns/op
BenchmarkTypeFieldsCache/HitTypes10-32 500000000 63.6 ns/op

@ALTree
Copy link
Member Author

ALTree commented May 31, 2018

I looked into this yesterday and I got nowhere. I'll send a revert for that change.

@gopherbot
Copy link

Change https://golang.org/cl/115535 mentions this issue: Revert "testing: only compute b.N once when passed -count > 1"

@ALTree ALTree mentioned this issue May 31, 2018
Open
@ALTree
Copy link
Member Author

ALTree commented May 31, 2018

The change was reverted. @laboger @erifan if you could verify that the bench regression is gone on ppc64le and arm64 that would be great.

@laboger
Copy link
Contributor

laboger commented May 31, 2018

I noticed this went in and tried it right away -- looks like it is now working as expected for ppc64le.

@erifan
Copy link

erifan commented Jun 4, 2018 •
edited

It is now working as expected for arm64.

@golang golang locked and limited conversation to collaborators Jun 4, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Projects
None yet
Milestone
Go1.11
Development

No branches or pull requests
4 participants
@ALTree @gopherbot @laboger @erifan