Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: Benchmark reporting incorrect stats when running in parallel #28813

Open
mnewswanger opened this issue Nov 15, 2018 · 4 comments
Open
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@mnewswanger
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.11 linux/amd64

Applies to earlier versions as well

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build896591388=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Using the following test file:

package bechmarkperf

import (
	"strconv"
	"testing"
	"time"
)

func BenchmarkSleepSerial(b *testing.B) {
	for i := 0; i < b.N; i++ {
		sleepFunc()
	}
}

func BenchmarkSleepParallel(b *testing.B) {
	for i := 1; i <= 8; i *= 2 {
		b.Run(strconv.Itoa(i), func(b *testing.B) {
			b.SetParallelism(i)
			b.RunParallel(func(pb *testing.PB) {
				for pb.Next() {
					sleepFunc()
				}
			})
		})
	}
}

func sleepFunc() {
	time.Sleep(1 * time.Millisecond)
}

Running in parallel, results were ~1ms / for ns/op. This mean that doubling threadcount would halve ns/op.

What did you expect to see?

ns/op hold the same and not go below 1ms.

What did you see instead?

When running with 128 goroutines, go the following output:

BenchmarkSleepParallel/8-16  	  100000	     12024 ns/op	       0 B/op	       0 allocs/op

This is reporting that the benchmark process is taking well under 1ms (which is the sleep duration above).

@gopherbot
Copy link

Change https://golang.org/cl/149720 mentions this issue: testing: correct benchmark output when running in parallel

@ianlancetaylor ianlancetaylor changed the title Benchmark reporting incorrect stats testing: Benchmark reporting incorrect stats when running in parallel Nov 16, 2018
@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 16, 2018
@ianlancetaylor ianlancetaylor added this to the Go1.13 milestone Nov 16, 2018
@odeke-em
Copy link
Member

Hello @mnewswanger, kindly paging you since Go1.13 is now being actively developed against, and Ian left some feedback on your CL. Please take a look. Thank you.

@mnewswanger
Copy link
Author

I've addressed Ian's comments (re: adding a test to cover the scenario), and I just rebased to resolve merge conflicts against master. Let me know if you need anything else there.

@mnewswanger
Copy link
Author

Anything else needed from me at this point?

@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants