Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing,go/doc: no way to properly distribute non-trivial runnable examples #36586

Open
mvdan opened this issue Jan 15, 2020 · 2 comments
Open
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@mvdan
Copy link
Member

mvdan commented Jan 15, 2020

I filed #35852 as a proposal to try to describe this problem and propose a solution, but since it was declined, I'm falling back to simply filing an issue about the problem. The fact that the solution got rejected doesn't mean that the problem doesn't exist.

--

We can have examples which show up in the godoc but aren't runnable, and runnable examples which are run as part of go test. This is enough for most use cases.

However, take the case where one has more than a handful of examples in a single package, and they each take a non-trivial amount of time to run, such as 200ms. Since they cannot run in parallel with themselves or any other test within the same package, go test can take multiple more seconds than it really needs to.

The root of the problem is that there's no way to write examples which are:

  1. Part of the documentation (i.e. godoc)
  2. Runnable (i.e. on godoc.org, or via go test)
  3. Scalable (i.e. can be run in parallel)

Below are some workarounds I considered:

  • Rewriting some of the examples as tests. Not good, because they are lost in the godoc, and hidden from the users. Loses point 1.
  • Splitting the examples in multiple packages. This makes go test ./... faster, as there's built-in parallelism between packages being tested. However, this splits up godoc pages, and it's not unreasonable to have more than a handful of examples in a single package. Loses point 1.
  • Make some of the examples non-runnable. Perhaps the best option today, but still not great. A runnable example is always verified by go test, and it's easier for the user to see what it does and how to run it. Loses point 2.

In the proposal above I tried exploring a couple of solutions to allow running examples in parallel. Are there other solutions we can look into?

/cc @bcmills @leitzler @rsc

@josharian
Copy link
Contributor

Examples must be run serially. But perhaps examples could be run concurrently with tests?

That’d surely break someone’s tests.

But maybe that could be done manually, if testing.M had RunExamples and RunTests (and I guess you’d need RunBenchmarks). Then people could fire up two goroutines, one for examples and one for tests.

@mvdan
Copy link
Member Author

mvdan commented Jan 16, 2020

Maybe - that's an interesting idea. It won't scale terribly well, but it would be far better than what we have now.

I'm not just fixated on ways to make the current way faster, by the way. That's what I tried with the proposal that got rejected. For example: maybe we should have an entirely new standard way to provide example code or packages, when those are too slow or complex to have directly as runnable examples. Being a standard is important, because you want software like godoc to show it to the user.

@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 21, 2020
@toothrot toothrot added this to the Backlog milestone Jan 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

3 participants