Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: add summary at end of go test results #30507

Closed
markvincentcaro opened this issue Mar 1, 2019 · 16 comments
Closed

cmd/go: add summary at end of go test results #30507

markvincentcaro opened this issue Mar 1, 2019 · 16 comments
Labels
FrozenDueToAge help wanted NeedsFix The path to resolution is known, but the work has not been done. Proposal Proposal-Accepted
Milestone

Comments

@markvincentcaro
Copy link

markvincentcaro commented Mar 1, 2019

Currently, there is no way to quickly tell if a test failure occurred after running go test.

You have to scroll up and look for the word FAIL or search for it in a text editor.

Printing a summary of the test results at the bottom (e.g. Tests run: 10, Failures: 2, Errors: 0, Skipped: 0) would make it easy to tell at a glance if any test failures occurred or not.

An answer to Issue #27755 suggested using the -json flag, but I haven't seen any good documentation or examples on how to use it. Also, this approach seems too convoluted for something that other languages have for tests out of the box.

EDIT: I'm on Windows 10.

@gopherbot gopherbot added this to the Proposal milestone Mar 1, 2019
@markvincentcaro markvincentcaro changed the title Proposal: cmd/go test - Add Test Summary at End of Results proposal: cmd/go test - Add Test Summary at End of Results Mar 1, 2019
@markvincentcaro markvincentcaro changed the title proposal: cmd/go test - Add Test Summary at End of Results proposal: cmd/go test - Add Summary at End of Test Results Mar 1, 2019
@mvdan
Copy link
Member

mvdan commented Mar 1, 2019

Currently, there is no way to quickly tell if a test failure occurred after running go test

The very last line should tell you. Even if you're testing many packages, each package that succeeded should only output a single line, so I don't see how this would be a problem unless you were testing a thousand packages at once.

Or, if you simply want to tell if any test failed, just check go test's exit code.

@markvincentcaro
Copy link
Author

markvincentcaro commented Mar 1, 2019

@mvdan

Even if you're testing many packages, each package that succeeded should only output a single line, so I don't see how this would be a problem unless you were testing a thousand packages at once.

Actually, the fact that you need to manually check if any package failed can be subject to human errors, however rare they may be. And you shouldn't be discouraged from decoupling your code into organized packages and writing tests for them just because you want to keep your package count low to make test verification easier. The compiler's tools should be doing that for you.

Or, if you simply want to tell if any test failed, just check go test's exit code.

How exactly do you do this?

@ianlancetaylor ianlancetaylor changed the title proposal: cmd/go test - Add Summary at End of Test Results proposal: cmd/go: add summary at end of go test results Mar 1, 2019
@ianlancetaylor
Copy link
Contributor

Or, if you simply want to tell if any test failed, just check go test's exit code.

How exactly do you do this?

What system are you running on? GNU/Linux? Windows?

@ianlancetaylor
Copy link
Contributor

Programmatically you check the exit code by checking whether os/exec's Run method return an error or not.

@mvdan
Copy link
Member

mvdan commented Mar 1, 2019

On unix-y shells, you can do go test whatever-packages; echo $? to see the last command's exit status. I'm sure Windows and all other systems will have similar mechanisms.

@mvdan mvdan added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 1, 2019
@markvincentcaro
Copy link
Author

markvincentcaro commented Mar 1, 2019

@mvdan
What info do you need? If it's my system, I added it in the description.

@mvdan
Copy link
Member

mvdan commented Mar 1, 2019

Well, @ianlancetaylor asked a question, and we both showed that it's generally simple to check an exit status code. You were asking how to quickly tell if a go test run had any failures - that's the simplest way to check. We're waiting for info because your entire proposal seems to be based around that problem.

@markvincentcaro
Copy link
Author

markvincentcaro commented Mar 2, 2019

@mvdan

The answer from this SO thread (https://stackoverflow.com/a/334890/7824245) says the errorlevel environment variable is how to check it on Windows, but to be safe, it also needs set errorlevel= to clear any previous values.

I'll try it out and see how that works.

@markvincentcaro
Copy link
Author

markvincentcaro commented Mar 5, 2019

@mvdan @ianlancetaylor

Nevermind, I found a better way to quickly detect test failures from this SO thread:

go test ./... -v -short > test-results.txt & findstr "FAIL _test" test-results.txt

Though I still kinda wish that the Go team would officially add this feature in the future.

@rsc
Copy link
Contributor

rsc commented Mar 7, 2019

Note that #27755 was about a per-package summary. This issue is about a per-run summary over many packages. In general I'm skeptical of summaries: totaling numbers across tests is misleading in various ways, and people won't agree about the level of detail. But I do sympathize with seeing the end of output and not even knowing whether the overall list succeeded.

I propose that in 'package list mode' (see 'go help test' for 'local directory mode' vs 'package list mode'), the overall 'go test' could end by printing FAIL when there are failures (but stay silent for all "ok"/"skip").

@markvincentcaro
Copy link
Author

markvincentcaro commented Mar 7, 2019

@rsc
Now this I can get behind. It would also be very helpful to put all failed tests, the specific line number, and t.Error messages at the bottom so we don't have to hunt for them at the top.

@andybons
Copy link
Member

Per discussion with @golang/proposal-review, let's move forward with @rsc's idea.

@andybons andybons added Proposal-Accepted and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Mar 27, 2019
@andybons andybons changed the title proposal: cmd/go: add summary at end of go test results cmd/go: add summary at end of go test results Mar 27, 2019
@ianlancetaylor ianlancetaylor added help wanted NeedsFix The path to resolution is known, but the work has not been done. labels Mar 27, 2019
@ianlancetaylor ianlancetaylor modified the milestones: Proposal, Go1.13 Mar 27, 2019
@gopherbot
Copy link

Change https://golang.org/cl/170948 mentions this issue: cmd/go: print finally FAIL if a test has failed in package list mode

@manikesh
Copy link

manikesh commented May 7, 2019

what is the resolution to this issue?? do we have any command which gives result in asked manner..

@ianlancetaylor
Copy link
Contributor

This issue is not yet resolved.

luka-zitnik added a commit to luka-zitnik/go that referenced this issue May 11, 2019
Fixes golang#30507

Change-Id: Ic598e4d5f71c624fcde051982bf85533e2f18e8d
luka-zitnik added a commit to luka-zitnik/go that referenced this issue May 12, 2019
Fixes golang#30507

Change-Id: Ic598e4d5f71c624fcde051982bf85533e2f18e8d
@marco-m
Copy link

marco-m commented Jun 1, 2019

@markvincentcaro check out https://github.com/gotestyourself/gotestsum

@golang golang locked and limited conversation to collaborators May 31, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge help wanted NeedsFix The path to resolution is known, but the work has not been done. Proposal Proposal-Accepted
Projects
None yet
Development

No branches or pull requests

8 participants