Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testing: unexpected top level test output belonging to a subtest #29755

Closed
mfridman opened this issue Jan 15, 2019 · 3 comments
Closed

testing: unexpected top level test output belonging to a subtest #29755

mfridman opened this issue Jan 15, 2019 · 3 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.15

Comments

@mfridman
Copy link

mfridman commented Jan 15, 2019 •
edited

What version of Go are you using (go version)?

go version devel +76e8ca4 Tue Jan 15 18:14:54 2019 +0000 darwin/amd64

Does this issue reproduce with the latest release?

N/A, working with master.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOOS="darwin"

What did you do?

Run code from this playground example

go test debug/golang/subtest -run ^TestThis$ -v -race

What did you expect to see?

Expected to see the test output

some_test.go:14: woah, something went wrong

belong to the top level test and not the inner subtest. E.g., when running the above with -json I'd expect this output:

{"Time":"2019-01-15T13:40:03.616451-05:00","Action":"output","Package":"debug/golang/subtest","Test":"TestThis/topLevelTest","Output":"        some_test.go:14: woah, something went wrong\n"}

note the test name in the json output is expected to beTestThis/topLevelTest

What did you see instead?

Instead the output and -json output makes it seem as if it belongs to the inner subtest, i.e., test name TestThis/topLevelTest/subA

This may be a misuse of the API, but I couldn't find any docs that mention there shouldn't be anything in between subtests.

=== RUN   TestThis
=== RUN   TestThis/topLevelTest
=== RUN   TestThis/topLevelTest/subA
--- FAIL: TestThis (0.00s)
    --- FAIL: TestThis/topLevelTest (0.00s)
        --- PASS: TestThis/topLevelTest/subA (0.00s)
            some_test.go:11: subA passing
        some_test.go:14: woah, something went wrong
FAIL
FAIL	debug/go-trun	0.026s

The -json output was:

{"Time":"2019-01-15T13:40:03.616451-05:00","Action":"output","Package":"debug/golang/subtest","Test":"TestThis/topLevelTest/subA","Output":"        some_test.go:14: woah, something went wrong\n"}

This might be an issue with test2json.
@mfridman mfridman changed the title testing: unexpected outer test output belonging to a subtest testing: unexpected top level test output belonging to a subtest Jan 15, 2019
@bcmills
Copy link
Contributor

bcmills commented Jan 29, 2019 •
edited

CC @mpvl @josharian

As far as I can tell the indentation looks right. I suspect a test2json bug.

@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 29, 2019
@bcmills bcmills added this to the Go1.13 milestone Jan 29, 2019
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@dnephin dnephin mentioned this issue Sep 16, 2019
Closed
@dnephin
Copy link
Contributor

dnephin commented Sep 18, 2019 •
edited

I ran into this bug as well. I started reading test2json for the first time.

From my not-yet-informed first read, it seems like the bug may be that converter.testName is used as event.Test. That value is a single field, not a stack, so when indentation drops back down the testName isn't being reset. converter.report seems to be a stack of event, which has a Name field.

Maybe testName could be removed and the value could be ready from report[-1]? (or we could add a new stack for testName if that won't work for some reason).

Anyone familiar with this code that might be able to confirm?

I can work on contributing a fix.

@dnephin dnephin mentioned this issue Sep 19, 2019
Closed
@gopherbot
Copy link

Change https://golang.org/cl/196617 mentions this issue: cmd/test2json: attribute output to the correct test

@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
dnephin added a commit to dnephin/gotestsum that referenced this issue Apr 19, 2020
@dnephin
store test output by rootTestName->subTestNames->output
30d9d3f 
In support of a workaround for golang/go#29755
dnephin added a commit to dnephin/gotestsum that referenced this issue Apr 19, 2020
@dnephin
Workaround test2json bug causing missing test output
74004d0 
See golang/go#29755.

If a test fails, and there is no test output (other than framing), then
print all the test output for the entire test case.

This won't fix the issue if the test which failed had some output, but
well behaved tests generally shouldn't non-failure message output, so
hopefully this case is rare.
@dnephin dnephin mentioned this issue Apr 19, 2020
Merged
dnephin added a commit to dnephin/gotestsum that referenced this issue Apr 19, 2020
@dnephin
Workaround test2json bug causing missing test output
aa9ed6b 
See golang/go#29755.

If a test fails, and there is no test output (other than framing), then
print all the test output for the entire test case.

This won't fix the issue if the test which failed had some output, but
well behaved tests generally shouldn't non-failure message output, so
hopefully this case is rare.
dnephin added a commit to dnephin/gotestsum that referenced this issue Apr 20, 2020
@dnephin
store test output by rootTestName->subTestNames->output
0c77b07 
In support of a workaround for golang/go#29755
dnephin added a commit to dnephin/gotestsum that referenced this issue Apr 20, 2020
@dnephin
summary: mitigate test2json bug causing missing test output
c2cbea1 
See golang/go#29755.

If a test fails, and there is no test output (other than framing), then
print all the test output for the entire test case.

This won't fix the issue if the test which failed had some output, but
well behaved tests generally shouldn't non-failure message output, so
hopefully this case is rare.
dnephin added a commit to dnephin/gotestsum that referenced this issue Apr 21, 2020
@dnephin
store test output by rootTestName->subTestNames->output
1233c41 
In support of a workaround for golang/go#29755
dnephin added a commit to dnephin/gotestsum that referenced this issue Apr 21, 2020
@dnephin
summary: mitigate test2json bug causing missing test output
6733193 
See golang/go#29755.

If a test fails, and there is no test output (other than framing), then
print all the test output for the entire test case.

This won't fix the issue if the test which failed had some output, but
well behaved tests generally shouldn't non-failure message output, so
hopefully this case is rare.
dnephin added a commit to dnephin/gotestsum that referenced this issue Apr 21, 2020
@dnephin
summary: mitigate test2json bug causing missing test output
cacb835 
See golang/go#29755.

If a test fails, and there is no test output (other than framing), then
print all the test output for the entire test case.

This won't fix the issue if the test which failed had some output, but
well behaved tests generally shouldn't non-failure message output, so
hopefully this case is rare.
@bcmills bcmills modified the milestones: Backlog, Go1.15 Jun 1, 2020
@bcmills bcmills added the NeedsFix The path to resolution is known, but the work has not been done. label Jun 1, 2020
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 1, 2020
@golang golang locked and limited conversation to collaborators Jun 1, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.15
Development

Successfully merging a pull request may close this issue.

cmd/test2json: attribute output to the correct test dnephin/go
6 participants
@rsc @andybons @dnephin @bcmills @mfridman @gopherbot