Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: fail if the same test function is defined in both the _test and non-_test packages #60977

Open
bcmills opened this issue Jun 23, 2023 · 3 comments
Labels
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Jun 23, 2023

(This proposal prompted by https://go-review.googlesource.com/c/go/+/488855/10//COMMIT_MSG#22.)

If a Go package has both an internal test (package example) and an external test (package example_test), the resulting test binary runs the Test, Fuzz, Example, and Benchmark functions from both packages.

If a TestMain function is defined in both packages, test setup fails with multiple definitions of TestMain.

However, if other testing functions are defined in both packages, they are both run. That leads to a number of confusing effects:

  1. The same test name can be shown with two different results (say, PASS and SKIP, or PASS and FAIL), even if the test flags only request to run each package once. That may confuse tools that ingest test results.
  2. If the tests run in parallel, logs and JSON events for the two tests may be interleaved, with no way to distinguish the two. (t.Log messages will list the correct file and line number, but log messages originating in helper functions may be completely ambiguous.)
  3. It is impossible to use the -run flag to run only one of the two tests but not the other.

For the above reasons, I propose that a test binary that defines two different Test, Fuzz, Example, or Benchmark functions with the same name should fail at run-time.

(A GODEBUG setting should be used to restrict this failure to only tests whose go version is at least the release in which this change is introduced.)

@gopherbot gopherbot added this to the Proposal milestone Jun 23, 2023
@bcmills
Copy link
Contributor Author

bcmills commented Jun 23, 2023

Alternately, this could be implemented as a vet check, included in the subset of vet checks run by default during go test.

However, I prefer the testing approach: the cmd/vet alternative seems more difficult to implement (especially in a way that complies with #56986), and either way it would cause go test of the package to fail.

@bcmills bcmills changed the title proposal: testing: fail the test if a test function is defined in both the _test and non-_test package with the same name proposal: testing: fail if the same test function is defined in both the _test and non-_test packages Jun 23, 2023
@ianlancetaylor
Copy link
Contributor

Why fail at runtime rather than failing when cmd/go generates the _testmain.go file?

@bcmills
Copy link
Contributor Author

bcmills commented Jun 23, 2023

Failing at run time allows a user to revert to the prior behavior easily using a //go:debug directive. Failing at generation time would make it a hard break for any existing duplicated tests when updating the go version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Incoming
Development

No branches or pull requests

3 participants