Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: support explicitly running non-hermetic (or non-deterministic) examples #31310

Open
mvdan opened this issue Apr 6, 2019 · 9 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@mvdan
Copy link
Member

mvdan commented Apr 6, 2019

We can have examples which show up in the godoc but aren't runnable, and runnable examples which are run as part of go test.

What about examples which are too slow or unreliable for go test, for example those needing access to the internet? I can make them non-runnable, but what if a user wants to give one a try? go test -run ExampleSlow doesn't work, and the next best option is copy-pasting into a main func, which is awkward.

The only other option, which is what I tend to do, is to stick examples as main packages elsewhere. But that's also icky; if the example packages aren't hidden, go install ./... will include them, and if they're hidden in another repo or in _examples, they are hard to find.

Another disadvantage of hiding the examples in a separate dir/repo is that go test ./... won't check if they compile fine; one has to remember to check that dir/repo too.

For some context, I'm trying to collapse the https://github.com/chromedp/examples into the main repo, starting with an example_test.go file. The second and third examples are quick, but the first accesses the internet and is slow. I don't want to run it with go test, but I want it to show up in godoc and allow people to run it easily to try it out.

I'm not labelling this as a proposal, because I don't have a specific solution in mind; this is just a description of a problem I have when trying to document packages with non-trivial examples.

/cc @acln0 @mpvl @bcmills @ianlancetaylor

@mvdan mvdan added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Apr 6, 2019
@mvdan mvdan changed the title testing: better support for slow/disabled examples testing: better support for slow or explicitly run examples Apr 6, 2019
@mvdan
Copy link
Member Author

mvdan commented Apr 6, 2019

One change that comes to mind is adding a testing flag, so that go test doesn't run a slow example, but an explicit command like go test -slowexamples -run SlowExample does. That's probably a terrible solution, but it helps illustrate the two alternatives I'd like to have available somehow.

Edit: a slightly nicer flag would be go test -allexamples -run ExampleWithInternet.

@beoran
Copy link

beoran commented Apr 8, 2019

You could probably about add a //build:slowexamples tag to those slow examples and then run them with go test -tag slowexamples -run "ExampleSlow.*". Would that solve your problem?

@mvdan
Copy link
Member Author

mvdan commented Apr 8, 2019

No; hiding the examples behind a build tag has the same issues as hiding them in an _examples package. They wouldn't show up in the godoc, and go test ./... wouldn't error if they can't build properly.

@bcmills
Copy link
Contributor

bcmills commented Apr 8, 2019

What about examples which are too slow or unreliable for go test, for example those needing access to the internet?

See previously the discussion on #30595, particularly #30595 (comment).

Unfortunately, unlike tests, runnable examples do not have a Short method to interrogate, nor a Skip method they can use to abort the example if it cannot be run. (They either have an Output line and always run, or they lack an Output line and always merely build.)

@bcmills bcmills added this to the Unplanned milestone Apr 8, 2019
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 8, 2019
@gopherbot gopherbot removed the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Apr 8, 2019
@mvdan
Copy link
Member Author

mvdan commented Apr 8, 2019

Thanks. I did see that thread, and I agree with Ian's response there. I don't think the same applies here; tests and examples are essentially different things. Generally, the users will be interested in the examples and will want to see and run them, regardless of whether they qualify as "slow" or "integration" examples.

@bcmills
Copy link
Contributor

bcmills commented Apr 8, 2019

I think there is a big difference between “see” and “run”.

I would argue that:

  • go doc should show all examples in the documentation, regardless of their speed or requirements.
  • go test should (by default) test only the subset of examples that are hermetic, leaving it up to users to explicitly enable the non-hermetic tests (by setting an environment variable or build tag).
  • go test -short should test only the subset of examples that are both hermetic and fast.

The connection to Short and Skip is that we currently have no way to mark an example as “slow”, and no way to check only compilation for examples that should be runnable but not run by default.

@mvdan
Copy link
Member Author

mvdan commented Apr 8, 2019

What you said makes sense to me, for the most part. The only note I'd add is that I don't care about the speed of examples per se; I've rarely encountered an example which took a lot of resources or time. I should have probably titled this issue with "non-hermetic examples" instead of "slow examples". Or perhaps even "non-deterministic examples".

@thepudds
Copy link
Contributor

thepudds commented Apr 8, 2019

I should have probably titled this issue with "non-hermetic examples" instead of "slow examples". Or perhaps even "non-deterministic examples".

If that is the focus, then there might be some relationship to this comment from Russ in #23799 (comment)

It would be OK to have a 'mark this as depending on external things' as long as that result was still cached by default, and then we could also add a 'rerun all the tests that depend on external things' as an explicit flag. Not sure what that would look like exactly.

Leaving for Go 1.13.

@mvdan mvdan changed the title testing: better support for slow or explicitly run examples testing: support explicitly running non-hermetic (or non-deterministic) examples Apr 8, 2019
@mvdan
Copy link
Member Author

mvdan commented Apr 8, 2019

Ok, I've retitled the issue to what I think is a better summary after the initial discussion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants