Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: cannot test multiple sub-benchmarks #47800

Closed
firelizzard18 opened this issue Aug 18, 2021 · 7 comments
Closed

cmd/go: cannot test multiple sub-benchmarks #47800

firelizzard18 opened this issue Aug 18, 2021 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@firelizzard18
Copy link
Contributor

What version of Go are you using (go version)?

$ go version
go version go1.17 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOCACHE="/home/ME/.cache/go-build"
GOENV="/home/ME/.config/go/env"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOMODCACHE="/home/ME/go/pkg/mod"
GOOS="linux"
GOPATH="/home/ME/go"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ME/sdk/go1.17"
GOSUMDB="sum.golang.org"
GOTOOLDIR="/home/ME/sdk/go1.17/pkg/tool/linux_amd64"
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/ME/src/runway/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3678785145=/tmp/go-build -gno-record-gcc-switches"

What did you do?

package bench_test

import "testing"

func BenchmarkFoo(b *testing.B) {
	b.Run("Bar", func(b *testing.B) {})
	b.Run("Baz", func(b *testing.B) {})
	b.Run("Bat", func(b *testing.B) {})
}
go test -run=- -bench='^(BenchmarkFoo/Bar|BenchmarkFoo/Baz)$'`

What did you expect to see?

Go executes the sub-benchmarks that match the pattern.

What did you see instead?

Go runs nothing:

PASS
ok      gitlab.com/eukano/runway        0.001s
@gopherbot
Copy link

Change https://golang.org/cl/343433 mentions this issue: src/goTestExplorer: support running sub-tests

@mknyszek
Copy link
Contributor

Here's a workaround:

benchtest $ go test -bench='^BenchmarkFoo/(Baz|Bar)$' -count=4 -v
goos: darwin
goarch: amd64
pkg: test
cpu: Intel(R) Core(TM) i7-7567U CPU @ 3.50GHz
BenchmarkFoo
BenchmarkFoo/Bar
BenchmarkFoo/Bar-4         	1000000000	         0.0000001 ns/op
BenchmarkFoo/Bar-4         	1000000000	         0.0000002 ns/op
BenchmarkFoo/Bar-4         	1000000000	         0.0000001 ns/op
BenchmarkFoo/Bar-4         	1000000000	         0.0000001 ns/op
BenchmarkFoo/Baz
BenchmarkFoo/Baz-4         	1000000000	         0.0000002 ns/op
BenchmarkFoo/Baz-4         	1000000000	         0.0000001 ns/op
BenchmarkFoo/Baz-4         	1000000000	         0.0000002 ns/op
BenchmarkFoo/Baz-4         	1000000000	         0.0000002 ns/op
PASS
ok  	test	0.114s

I think this is working as intended though. Running go help testflag reveals:

	-bench regexp
	    Run only those benchmarks matching a regular expression.
	    By default, no benchmarks are run.
	    To run all benchmarks, use '-bench .' or '-bench=.'.
	    The regular expression is split by unbracketed slash (/)
	    characters into a sequence of regular expressions, and each
	    part of a benchmark's identifier must match the corresponding
	    element in the sequence, if any. Possible parents of matches
	    are run with b.N=1 to identify sub-benchmarks. For example,
	    given -bench=X/Y, top-level benchmarks matching X are run
	    with b.N=1 to find any sub-benchmarks matching Y, which are
	    then run in full.

Note that the regexp is split by "/", hence why it doesn't work. I admit this is somewhat hidden, though. Closing for now, we should probably have another issue about surfacing this better, somehow, if it doesn't exist yet.

@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 19, 2021
@mknyszek mknyszek added this to the Backlog milestone Aug 19, 2021
@mknyszek mknyszek changed the title cmd/test: cannot test multiple sub-benchmarks cmd/go: cannot test multiple sub-benchmarks Aug 19, 2021
@seankhliao
Copy link
Member

@mknyszek closed on purpose or accident?

@mknyszek
Copy link
Contributor

On purpose. See the end of my last message.

@seankhliao
Copy link
Member

Ah , should be dup of #39904

@firelizzard18
Copy link
Contributor Author

@mknyszek I’m working on vscode-go, so I need an automated way of doing this. What I really need is a flag that lets me specify a list of tests and sub tests. The regex syntax is problematic for this case.

@mknyszek
Copy link
Contributor

@seankhliao Thanks for finding that!

@firelizzard18 Looking at it briefly, it appears that #39904 is close to what you want (maybe even exactly), and it looks like that proposal was accepted a year ago. AFAICT it just needs someone to implement it.

gopherbot pushed a commit to golang/vscode-go that referenced this issue Sep 3, 2021
Changes the logic for creating sub-tests on the fly. Prevously, test
IDs were created in such a way that it was impractical to support
running subtests. This changes the way IDs are created for subtests
such that running subtests is simple.

Additionally, this CL updates 'goTest' to run `go test -run=^$ -bench
^BenchmarkFoo/Bar$` (without the parentheses) when a single benchmark
is selected, as a hack to get around golang/go#47800.

Updates #1641

Change-Id: I26eac8a5a396df3923073274ed93d9c59107d9c3
Reviewed-on: https://go-review.googlesource.com/c/vscode-go/+/343433
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
Trust: Hyang-Ah Hana Kim <hyangah@gmail.com>
Trust: Rebecca Stambler <rstambler@golang.org>
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
@golang golang locked and limited conversation to collaborators Aug 19, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants