For what it's worth, I have wanted a very similar thing for go list -json and go list -m -json; they both emit static structures, and their types are only documented in go help list. Right now, one has to copy those types (or the subset of fields one needs) into Go code, as there is no readily available package for them.

There are also a multitude of other similar -json flags where the output follows a struct: go mod edit -json, go work edit -json, go env -json. And there will likely be more in the future, like #34293.

All this to say: I think we should have a package or module to contain all of these, and not just go test -json. I don't mean to block the proposal or its implementation until we have all of them, but at least we shouldn't name the package anything too specific like gotest - perhaps x/tools/cmdgo, with types like TestOutput and Module.

We could publish a parser but it would be even better to fix the go command to make -json print only JSON.
Right now I think JSON is on stdout and non-JSON is on stderr. Is that right? Or is the situation worse than that?

/cc @aclements who has been looking at JSON output in all.bash.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc unfortunately there is some non-JSON printed to stdout, intermingled with the JSON. This is a little test module I put together to examine the various permutations:

~/src/test-output $ for f in $(find * -type f); do echo "--- $f"; cat $f; done
--- buildfail/nobuild_test.go
package main

import "testing"

func TestHelloWorld(t *testing.T) {
	var s string = t
}
--- go.mod
module test-output

go 1.18
--- notest/hello.go
package notest

import "fmt"

var Println = fmt.Println
--- root_test.go
package main

import "testing"

func TestHelloWorld(t *testing.T) {
	// t.Fatal("not implemented")
}
--- setupfail/nobuild_test.go
package main

import "testing"

func TestHelloWorld(t *testing.T) {
	asdfasdfs=a=asdf
}
--- tests/tests_test.go
package main

import "testing"

func TestSkip(t *testing.T) {
	t.Skip("skippin")
}

func TestFail(t *testing.T) {
	t.Fatal("failin")
}

func TestPass(t *testing.T) {
	t.Log("passin")
}
~/src/test-output $ go test -json ./... 2> stderr > stdout
~/src/test-output $ cat stdout
FAIL	test-output/setupfail [setup failed]
{"Time":"2022-07-21T08:58:56.000673+10:00","Action":"run","Package":"test-output","Test":"TestHelloWorld"}
{"Time":"2022-07-21T08:58:56.000794+10:00","Action":"output","Package":"test-output","Test":"TestHelloWorld","Output":"=== RUN   TestHelloWorld\n"}
{"Time":"2022-07-21T08:58:56.000808+10:00","Action":"output","Package":"test-output","Test":"TestHelloWorld","Output":"--- PASS: TestHelloWorld (0.00s)\n"}
{"Time":"2022-07-21T08:58:56.000814+10:00","Action":"pass","Package":"test-output","Test":"TestHelloWorld","Elapsed":0}
{"Time":"2022-07-21T08:58:56.000821+10:00","Action":"output","Package":"test-output","Output":"PASS\n"}
{"Time":"2022-07-21T08:58:56.000826+10:00","Action":"output","Package":"test-output","Output":"ok  \ttest-output\t(cached)\n"}
{"Time":"2022-07-21T08:58:56.000831+10:00","Action":"pass","Package":"test-output","Elapsed":0}
FAIL	test-output/buildfail [build failed]
{"Time":"2022-07-21T08:58:56.001515+10:00","Action":"output","Package":"test-output/notest","Output":"?   \ttest-output/notest\t[no test files]\n"}
{"Time":"2022-07-21T08:58:56.00154+10:00","Action":"skip","Package":"test-output/notest","Elapsed":0}
{"Time":"2022-07-21T08:58:56.307002+10:00","Action":"run","Package":"test-output/tests","Test":"TestSkip"}
{"Time":"2022-07-21T08:58:56.307086+10:00","Action":"output","Package":"test-output/tests","Test":"TestSkip","Output":"=== RUN   TestSkip\n"}
{"Time":"2022-07-21T08:58:56.307115+10:00","Action":"output","Package":"test-output/tests","Test":"TestSkip","Output":"    tests_test.go:6: skippin\n"}
{"Time":"2022-07-21T08:58:56.307136+10:00","Action":"output","Package":"test-output/tests","Test":"TestSkip","Output":"--- SKIP: TestSkip (0.00s)\n"}
{"Time":"2022-07-21T08:58:56.307144+10:00","Action":"skip","Package":"test-output/tests","Test":"TestSkip","Elapsed":0}
{"Time":"2022-07-21T08:58:56.307155+10:00","Action":"run","Package":"test-output/tests","Test":"TestFail"}
{"Time":"2022-07-21T08:58:56.307161+10:00","Action":"output","Package":"test-output/tests","Test":"TestFail","Output":"=== RUN   TestFail\n"}
{"Time":"2022-07-21T08:58:56.307166+10:00","Action":"output","Package":"test-output/tests","Test":"TestFail","Output":"    tests_test.go:10: failin\n"}
{"Time":"2022-07-21T08:58:56.307172+10:00","Action":"output","Package":"test-output/tests","Test":"TestFail","Output":"--- FAIL: TestFail (0.00s)\n"}
{"Time":"2022-07-21T08:58:56.307179+10:00","Action":"fail","Package":"test-output/tests","Test":"TestFail","Elapsed":0}
{"Time":"2022-07-21T08:58:56.307185+10:00","Action":"run","Package":"test-output/tests","Test":"TestPass"}
{"Time":"2022-07-21T08:58:56.30719+10:00","Action":"output","Package":"test-output/tests","Test":"TestPass","Output":"=== RUN   TestPass\n"}
{"Time":"2022-07-21T08:58:56.307196+10:00","Action":"output","Package":"test-output/tests","Test":"TestPass","Output":"    tests_test.go:14: passin\n"}
{"Time":"2022-07-21T08:58:56.307202+10:00","Action":"output","Package":"test-output/tests","Test":"TestPass","Output":"--- PASS: TestPass (0.00s)\n"}
{"Time":"2022-07-21T08:58:56.307207+10:00","Action":"pass","Package":"test-output/tests","Test":"TestPass","Elapsed":0}
{"Time":"2022-07-21T08:58:56.307212+10:00","Action":"output","Package":"test-output/tests","Output":"FAIL\n"}
{"Time":"2022-07-21T08:58:56.307413+10:00","Action":"output","Package":"test-output/tests","Output":"FAIL\ttest-output/tests\t0.259s\n"}
{"Time":"2022-07-21T08:58:56.307429+10:00","Action":"fail","Package":"test-output/tests","Elapsed":0.259}
~/src/test-output $ cat stderr
# test-output/setupfail
setupfail/nobuild_test.go:6:13: expected '==', found '='
# test-output/buildfail [test-output/buildfail.test]
buildfail/nobuild_test.go:6:17: cannot use t (variable of type *testing.T) as type string in variable declaration

Ideally all output would be printed as JSON when the -json flag is set. The setup and build failures above—the specifics of those failures—are the most important information in my use case (and probably most people's).

Thanks @adg. /cc @bcmills for thoughts about how we should proceed here.

I agree that a good first step would be to fix go test -json to emit only JSON to stdout.

I think it would be fine to also publish a standard Go struct definition, but at that point the parser shouldn't need to be anything other than a standard JSON parser.

I think it would be fine to also publish a standard Go struct definition, but at that point the parser shouldn't need to be anything other than a standard JSON parser.

Following up on the thought to publish a package, we briefly talked about this on today's tools call. Some points raised between Bryan and myself:

• We should keep the package name generic, like cmdgo and not cmdgotypes, because in the future we may want to include other APIs related to interfacing with the cmd/go CLI, such as cmd/internal/quoted, needed to pass arguments via flags like -ldflags.
• The package probably shouldn't be part of the standard library, because these Go types would technically not be fully covered by the Go1 compatibility guarantee, like cmd/go itself. For example, one could imagine changing the go test -json Go types as long as they marshal and unmarshal to JSON in a compatible way.

cc @aclements for visibility, who has been writing tooling parsing test JSON output.

I would definitely consider emitting any non-json to be a bug that we should fix. Thanks for the reproducer example. I've been working on support for -json output from all.bash, and I suspect we'll have to fix that issue as part of that regardless.

Once the output is purely JSON, I'm not clear that a standard parser is especially valuable. The struct is quite simple (especially compared to, e.g., the types emitted by go list). I'm not opposed to putting that definition somewhere public, it just seems pretty minor. The more annoying problem I had when writing a test2json parser was dealing with the newline-delimited stream: while I could use encoding/json.Decoder to stream each object, at the end of the stream it couldn't distinguish between an EOF with no more JSON (which is expected and not an error) and an EOF after reading partial JSON (which is an error). I wound up having to read each line using a Scanner and use a non-streaming JSON decode of each line (and that raises questions of what the maximum line length is). Though even this was at most a minor annoyance.

Filed #54378 for fixing go test -json. I think we can probably hold this discussion until that's done.

The more annoying problem I had when writing a test2json parser was dealing with the newline-delimited stream

Actually, I take this back. This is easy to do with encoding/json.Decoder.More, but since I was working on cmd/dist, I couldn't use that method because it wasn't defined in Go 1.4. (Now that 1.20 requires 1.17 to bootstrap, I should be able to make this much simpler. 😄)

I agree that if #54378 is addressed then the proposed package is of little utility, or that it can live elsewhere outside the Go project itself. (I'll consider releasing my continuous build stuff as an open source project at some point.)

I'll close this issue. If #54378 doesn't satisfactorily address the overall issues then we can revisit. Thanks all!

Placed on hold.
— rsc for the proposal review group