If this file can't be created, for example, because the package is part of the read-only module cache,

Given the -modcacherw flag, we should probably avoid even trying to write to modules outside of the main module or workspace — it would be really confusing to see a recurring fuzz failure that doesn't reproduce on a clean machine.

Change https://golang.org/cl/359256 mentions this issue: cmd/go: test the interaction between the fuzz corpus and the module cache

I added a test illustrating the current behavior.

I see two or three options to fix the situation:

1. We could add a flag (perhaps -test.fuzzdir?) to allow the caller of the test binary to specify an input/output directory, which would be read in addition to the default testdata/fuzz directory, but written instead of the default directory. Then go test -fuzz=. could explicitly set that flag to redirect or suppress output when fuzzing a package loaded from outside the main module(s).

   • This would require another change to the testing.testDeps interface, which may break tools that make assumptions about that interface. (Tools are not supposed to make assumptions about that interface, but some still do anyway.)
   • If we take this approach, should an input found during go test -fuzz=. cause the test to fail when run without the -fuzz flag? It seems surprising either way: either the fuzz crash isn't repeatable, or the non-fuzz behavior of the test depends on the state of the fuzz cache (not just the contents of the test's own module).

2. We could add a flag (perhaps -test.fuzzsave=false?) to allow the caller of the test binary to suppress writes to the testdata directory, and have go test -fuzz=. set that flag.

   • This would reduce the usefulness of go test -fuzz outside the main module, but it would also be a bit simpler to explain and implement.

3. We could make go test -fuzz explicitly fail for packages outside of the main module, punting this question to a later Go release (but allowing any of the possible behaviors).

I think I prefer option (3) for Go 1.18. If we reject go test -fuzz on any package outside the main module with a clear error message, then we can decide between (1) and (2) in a later release when we have a bit more breathing room.

Change https://golang.org/cl/359414 mentions this issue: cmd/go: disallow the -fuzz flag for tests outside the main module

Another couple options, both which still allow fuzzing in a different module:

4. We could write the crash to the corpus cache instead (in $GOCACHE/fuzz). In the exit message, we can explain that fuzzing was run in a module outside the main one, and we didn't write the crash to testdata as a result.

5. We could not write the crash anywhere, and just print it to the terminal instead. That's probably not the best option though, because the contents of the file might be massive.

I'm fine with punting this to 1.19 though.

(4) is, I think, basically equivalent to my (1). (go test could by default set -test.fuzzdir to $GOCACHE/fuzz, but could allow it to be overridden as well.)

The main issue I foresee with that approach is that it makes reproducibility tricky: it would either cause go test example.com/fuzzfail (without -fuzz=.) to fail all of the time, even when run from a pristine other module, or cause it to fail only when run within a specific other module (if we include the identity of the main module as part of the cache key), or cause it not to fail at all after a fuzzing failure is reported (which would be a surprising difference vs. fuzzing within a module), and clearing out the module cache and/or build cache might also have the side-effect of changing the non--fuzz behavior of the test.

(5) is my (2), but stated more concisely. 🙂

Change https://golang.org/cl/359481 mentions this issue: cmd/go: consolidate fuzz-support checks

Change https://golang.org/cl/359574 mentions this issue: cmd/go: document that test must not write to their source modules