Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: add -failfast to stop after first test failure #21700

Closed
cespare opened this issue Aug 30, 2017 · 8 comments
Closed

testing: add -failfast to stop after first test failure #21700

cespare opened this issue Aug 30, 2017 · 8 comments

Comments

@cespare
Copy link
Contributor

cespare commented Aug 30, 2017

I propose we add a testing flag to cause go test to exit after the first test failure.

There are two cases where I've wanted such a flag:

  • When I'm debugging a flaky test, I want to run it until failure. I often use -count 10000 for this purpose, but it will keep running the test after it fails once, so if I'm not looking at my terminal while it's running I need to go hunt through my scrollback to locate the failure. (Alternatively, I can use go test -c and re-run the test until failure using my shell, but this is slower, particularly if there's some kind of setup in a TestMain.)
  • When I'm debugging a test that has many failures, I want to focus on and fix a single test at a time. Today I can pick one of the failures and use -run to select it, but then once I've fixed that failure I want to focus the next one so I need to run all the tests again and repeat the process. It would be more convenient to run go test -exit1 (need a better flag name).

In the presence of t.Parallel this gets a little trickier. I suggest handling it by collecting the parallel results as they arrive and then exiting after the first failure, printing only the results that came in before the failure and then the failure last, but not printing results that arrived after the failure. The main subtlety for the user is that they might see output logged from other parallel tests which don't have success/failure printouts, but that seems like a minor concern.

@gopherbot gopherbot added this to the Proposal milestone Aug 30, 2017
@AlexRouSg
Copy link
Contributor

AlexRouSg commented Aug 31, 2017

You could have a single test function and use subtests and then check if one failed and don't run the rest.

func TestFoo(t *testing.T) {
    failed := false

    if !failed {
        t.Run("A=1", func(t *testing.T) { failed = true })
    }

    if !failed {
        t.Run("A=2", func(t *testing.T) { failed = true })
    }

    if !failed {
        t.Run("A=3", func(t *testing.T) { failed = true })
    }
}

@cespare
Copy link
Contributor Author

cespare commented Sep 6, 2017

@AlexRouSg sure, or I can comment out all the other tests. This proposal is about a way to improve the use cases I mentioned without restructuring the code.

@rsc
Copy link
Contributor

rsc commented Oct 16, 2017

This seems OK but we have to come up with a decent flag name.

@spf13
Copy link
Contributor

spf13 commented Oct 16, 2017

In a few other similar tools the flag used is "--failfast"

  • Django
  • Rspec

@mvdan
Copy link
Member

mvdan commented Oct 20, 2017

Others that come to mind are -failstop and -firstfail.

@rsc
Copy link
Contributor

rsc commented Oct 20, 2017

Let's use -(test.)failfast. If anyone wants to send a CL, please go ahead. (Remember to write a test and run mkalldocs.sh.) Thanks.

@rsc rsc modified the milestones: Proposal, Go1.10 Oct 20, 2017
@rsc rsc changed the title proposal: testing: add a flag to exit after the first test failure testing: add -failfast to stop after first test failure Oct 20, 2017
@gopherbot
Copy link

Change https://golang.org/cl/74450 mentions this issue: testing: add -failfast to stop after first test failure

@inancgumus
Copy link
Contributor

inancgumus commented Nov 29, 2017

I completed my work on this. Currently, it tries to stop the tests from running. However, with parallel tests, this isn't easy to do right now (I guess).

@golang golang locked and limited conversation to collaborators Nov 29, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants