Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go test -bench=. with -benchtime=1x is timing out at 2m and not running a single iteration of the benchmark #71446

Closed
jonathan-innis opened this issue Jan 26, 2025 · 8 comments
Labels
BugReport Issues describing a possible bug in the Go implementation.

Comments

@jonathan-innis
Copy link

Go version

go version go1.23.2 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE='on'
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/joinnis/Library/Caches/go-build'
GOENV='/Users/joinnis/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/joinnis/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/joinnis/go'
GOPRIVATE=''
GOPROXY='direct'
GOROOT='/Users/joinnis/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.2.darwin-arm64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/joinnis/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.2.darwin-arm64/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.23.2'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/joinnis/Library/Application Support/go/telemetry'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/joinnis/github/karpenter/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/dt/p6bcmvf11cl4lfj4dwjzcg140000gr/T/go-build1360673948=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Running go test -bench=. -benchtime=1x continually times out at 120s with the following error. My guess is that the test is not able to fully complete an iteration of the test function in this time, but I can't figure out why it's always timing out at 120s and it isn't able to fully complete an iteration of the benchmark function like I have asked it to do.

=== RUN   TestSchedulingProfile
Running the benchmark with 10 pods
Running the benchmark with 100 pods
Running the benchmark with 500 pods
Running the benchmark with 1000 pods
Running the benchmark with 1500 pods
Running the benchmark with 2000 pods
Running the benchmark with 5000 pods
--- FAIL: TestSchedulingProfile (127.36s)
panic: runtime error: integer divide by zero [recovered]
	panic: runtime error: integer divide by zero

goroutine 114 [running]:
testing.tRunner.func1.2({0x10281d400, 0x103a69a30})
	/Users/joinnis/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.2.darwin-arm64/src/testing/testing.go:1632 +0x1bc
testing.tRunner.func1()
	/Users/joinnis/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.2.darwin-arm64/src/testing/testing.go:1635 +0x334
panic({0x10281d400?, 0x103a69a30?})
	/Users/joinnis/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.2.darwin-arm64/src/runtime/panic.go:785 +0x124
sigs.k8s.io/karpenter/pkg/controllers/provisioning/scheduling_test.TestSchedulingProfile(0x1400012c1a0)
	/Users/joinnis/github/karpenter/pkg/controllers/provisioning/scheduling/scheduling_benchmark_test.go:123 +0x714
testing.tRunner(0x1400012c1a0, 0x102abf950)
	/Users/joinnis/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0xe4
created by testing.(*T).Run in goroutine 1
	/Users/joinnis/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x314

What did you see happen?

The test benchmark function times-out indicating in results.N that it did not complete any iterations of the benchmark.

What did you expect to see?

I expected results.N to say that it completed a single iteration no matter how long this benchmark took.

@gabyhelp gabyhelp added the BugReport Issues describing a possible bug in the Go implementation. label Jan 26, 2025
@seankhliao
Copy link
Member

tests need to pass before benchmarks are run.
you aren't even reaching the benchmark stage.

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

@seankhliao seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale Jan 26, 2025
@jonathan-innis
Copy link
Author

jonathan-innis commented Jan 26, 2025

tests need to pass before benchmarks are run

Tests are failing due to an integer divide by zero that's happening due to the number of iterations that are running by the benchmarker (0). You can see the test is being run here where we are dividing by the number of iterations that we are executing, but the benchmark is not executing a run.

This strikes me as a bug if I've asked the benchmarker to execute one run and it isn't doing that.

@seankhliao
Copy link
Member

a test TestSchedulingProfile does not run the benchmarker , only benchmark functions do BenchmarkXxx.

@jonathan-innis
Copy link
Author

does not run the benchmarker

It calls testing.Benchmark within the function, see here

@seankhliao
Copy link
Member

I think you'll have to show a simplified reproducer, the code your pointing at doesn't match your output (line numbers don't match), and a simple test of testing.Benchmark in a test function doesn't support b.N being 0.

@jonathan-innis
Copy link
Author

It does look like I pointed to the wrong line. Here's the line that's actually throwing the error. I did do a Println on res.N and res.T in my environment and got 0 and 0s respectively. Still trying to get a toy example working but at first glance, those values definitely look suspicous.

@jonathan-innis
Copy link
Author

Oh! I think I figured it out, it's the b.Fatalf statement here. I was under the impression that this would cause a panic, but looks like this was just causing silent failure from the test that didn't quite point back to the actual cause.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BugReport Issues describing a possible bug in the Go implementation.
Projects
None yet
Development

No branches or pull requests

3 participants