Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/test2json: output is incorrect and fails to mark tests if using TestMain #49866

Closed
dsesami opened this issue Nov 29, 2021 · 3 comments
Closed
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@dsesami
Copy link

dsesami commented Nov 29, 2021

In summary, using the JSON output format seems to not function correctly when using TestMain().

Assume two tests are available:

  1. TestMain()
  2. TestSomethingElse()

TestMain() is reserved and triggers TestSomethingElse() with m.Run(). I noticed this when using a go test to JUnit conversion tool. Initially, I thought that the bug was in the JUnit conversion tool, but then noted that the actions outputted by the json flag (in other words go tool test2json) had changed when I added TestMain():

  1. If TestMain() passes, it does not get the "Action": "run" field, and it is missing the "Test": field in the JSON output. If it fails, it is registered as an individual test and has appropriate action fields.
  2. If TestSomethingElse() is called from TestMain, it is missing its "Test": field in its JSON entirely on pass, but registers actions correctly if it fails.

Without TestMain():

{
  "Time":"XYZ",
  "Action":"output",
  "Test":"TestSomethingElse",
  "Output":"XYZ Test logs foo bar baz"
}

With TestMain() calling it:

{
  "Time":"XYZ",
  "Action":"output",
  "Output":"XYZ Test logs foo bar baz"
}

  1. What version of Go are you using (go version)?

$ go version
1.17

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
amd64, gccgo, other not apparently relevant flags...

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2469022733=/tmp/go-build -gno-record-gcc-switches"

What did you do?

You can't actually run this in the code playground, as you need to run go tool test2json on a compiled go test binary (or use the -json flag)

Here's some test code you can paste into your environment, it's just TestMain() and a dummy test. https://go.dev/play/p/KDtlzPW9gDU

Compile it and run the binary as go tool test2json -t ./your_binary.

@mknyszek mknyszek added this to the Backlog milestone Nov 30, 2021
@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 30, 2021
@mknyszek mknyszek changed the title go tool test2json output is incorrect and fails to mark tests if using TestMain(). cmd/test2json: output is incorrect and fails to mark tests if using TestMain Nov 30, 2021
@mknyszek
Copy link
Contributor

CC @rsc as the primary person in the git blame for test2json since we don't have an entry in https://dev.golang.org/owners for it.

@seankhliao
Copy link
Member

TestMain doesn't affect this.
As indicated by the test2json help output::

usage: go tool test2json [-p pkg] [-t] [./pkg.test -test.v]

pass the -test.v flag to include the information on the surrounding tests

@dsesami
Copy link
Author

dsesami commented Nov 30, 2021

Looks like you're right about -test.v! However, it's super strange to me that this action field isn't added during non -test.v runs. All log lines within a test should have that "Test" field in its JSON.

@golang golang locked and limited conversation to collaborators Nov 30, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants