Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go/internal/test: go test -list doesn't always display all examples #21205

Closed
dmitshur opened this issue Jul 28, 2017 · 5 comments
Closed
Labels
FrozenDueToAge GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@dmitshur
Copy link
Contributor

Go 1.9 adds a new feature described at https://beta.golang.org/doc/go1.9#go-test-list:

The go test command accepts a new -list flag, which takes a regular expression as an argument and prints to stdout the name of any tests, benchmarks, or examples that match it, without running them.

go test -help says:

-list regexp
    List tests, benchmarks, or examples matching the regular expression.
    No tests, benchmarks or examples will be run. This will only
    list top-level tests. No subtest or subbenchmarks will be shown.

It's issue #17209, implemented in CL 41195 (ba8ff87). /cc @nemith @mpvl

What version of Go are you using (go version)?

go version go1.9rc1 darwin/amd64

What operating system and processor architecture are you using (go env)?

$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/Dmitri/Dropbox/Work/2013/GoLanding:/Users/Dmitri/Dropbox/Work/2013/GoLand"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/tw/kgz4v2kn4n7d7ryg5k_z3dk40000gn/T/go-build927351599=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

I ran go test -v and go test -list=Example on a sample package with 3 examples, two of which are runnable (i.e., they have a "Output:" comment at the end), and one is not runnable (no "Output:" comment):

package main_test

import "fmt"

func ExampleLoud() {
	// Dog says:
	fmt.Println("BARK!")

	// Output: BARK!
}

func ExampleQuiet() {
	// Gopher says:
	fmt.Println("")

	// Output:
}

func ExampleNoOutput() {
	// This example has no "Output:" comment at the end,
	// so it doesn't get run, it's just shown in docs and
	// ensured to build without errors.

	_ = fmt.Sprintln("*tree falling sound*")
}

Note, the above is just a minimal reproduce sample. I ran into this issue in a real Go package github.com/shurcooL/issuesapp/httpclient:

httpclient $ go test -v
=== RUN   ExampleNewIssues
--- PASS: ExampleNewIssues (0.00s)
=== RUN   ExampleNewIssues_authenticated
--- PASS: ExampleNewIssues_authenticated (0.00s)
=== RUN   ExampleIssues_List
--- PASS: ExampleIssues_List (0.00s)
=== RUN   ExampleIssues_Count
--- PASS: ExampleIssues_Count (0.00s)
=== RUN   ExampleIssues_ListComments
--- PASS: ExampleIssues_ListComments (0.00s)
PASS
ok  	github.com/shurcooL/issuesapp/httpclient	0.011s
httpclient $ go test -list=Example
ExampleIssues_List
ExampleIssues_Count
ExampleIssues_ListComments
ok  	github.com/shurcooL/issuesapp/httpclient	0.012s
httpclient $ 

What did you expect to see?

$ go test -v
=== RUN   ExampleLoud
--- PASS: ExampleLoud (0.00s)
=== RUN   ExampleQuiet
--- PASS: ExampleQuiet (0.00s)
PASS
ok  	sample/package	0.006s
$ go test -list=Example
ExampleLoud
ExampleQuiet
ok  	sample/package	0.005s
$ 

What did you see instead?

$ go test -v
=== RUN   ExampleLoud
--- PASS: ExampleLoud (0.00s)
=== RUN   ExampleQuiet
--- PASS: ExampleQuiet (0.00s)
PASS
ok  	sample/package	0.006s
$ go test -list=Example
ExampleLoud
ok  	sample/package	0.005s
$ 

ExampleQuiet is not listed in go test -list=Example output, even though it's a valid runnable example [1] that runs as part of go test -v.

[1] https://golang.org/cmd/go/#hdr-Description_of_testing_functions

Cause

The cause is simple and can be seen in the code implementing the feature at ba8ff87#diff-70e628298261565d825f7199d13042f2R972. testing.listTests function has this loop for printing examples:

for _, example := range examples {
	if ok, _ := matchString(*matchList, example.Name); ok && example.Output != "" {
		fmt.Println(example.Name)
	}
}

Notably, it checks that example.Output != "". I'm assuming it does that to try to find runnable examples. However, some runnable examples (i.e., ones with last comment that starts with "Output:" or "Unordered output:") have empty expected output.

They're still runnable, and should be included in output of go test -list, just like they are in output of go test -v.

@dmitshur dmitshur added the GoCommand cmd/go label Jul 28, 2017
@dmitshur dmitshur added this to the Go1.9 milestone Jul 28, 2017
@dmitshur
Copy link
Contributor Author

dmitshur commented Jul 28, 2017

Added Go1.9 milestone since this is a bug in a new Go 1.9 feature, feel free to update if needed.

@dmitshur dmitshur added the NeedsFix The path to resolution is known, but the work has not been done. label Jul 28, 2017
@gopherbot
Copy link

Change https://golang.org/cl/52030 mentions this issue: cmd/go/internal/test: Fix go test -list command to show all examples

@gopherbot
Copy link

Change https://golang.org/cl/52110 mentions this issue: cmd/go/internal/test: Fix go test -list command to show all examples

@nemith
Copy link
Contributor

nemith commented Jul 31, 2017

Yes the output check isn't needed. The check is already done when the testmain.go is generated.

https://github.com/golang/go/blob/master/src/cmd/go/internal/test/test.go#L1491

@dmitshur
Copy link
Contributor Author

dmitshur commented Jul 31, 2017

Yes the output check isn't needed. The check is already done when the testmain.go is generated.

if e.Output == "" && !e.EmptyOutput {

That's very helpful, thanks for pointing that out!

That means the distinction between empty and missing "Output:" comment is already there, and this fix is very simple (it won't need to build all that functionality). It's just a matter of removing the unneeded example.Output != "" condition.

@golang golang locked and limited conversation to collaborators Aug 2, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

3 participants