Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go/internal/cache: provide GOFUZZCACHE environment variable #48901

Open
AlekSi opened this issue Oct 10, 2021 · 6 comments
Open

cmd/go/internal/cache: provide GOFUZZCACHE environment variable #48901

AlekSi opened this issue Oct 10, 2021 · 6 comments
Labels
FeatureRequest fuzz Issues related to native fuzzing support NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone

Comments

@AlekSi
Copy link
Contributor

AlekSi commented Oct 10, 2021

Go provides GOCACHE and GOMODCACHE environment variables to control build and modules cache locations. There is no environment variable to control fuzz corpus cache location; it is hard-coded to be $GOCACHE/fuzz.

I think that fuzz cache is significantly different from the build (and test) cache:

For that reason, I think it would be helpful to provide GOFUZZCACHE environment variable to control fuzz cache location. That would help people building automation around sharing that cache between machines.

(I'm also not really sure that fuzz cache directory should be a subdirectory of the build cache by default, but that might be too late to change that.)

/cc @katiehockman @jayconrod

@AlekSi

This comment has been minimized.

@gopherbot gopherbot added the fuzz Issues related to native fuzzing support label Oct 10, 2021
@jayconrod
Copy link
Contributor

We had a GOFUZZCACHE environment variable early on, but we made a design decision to remove it. We didn't feel like it added enough value to justify the extra complexity in the CLI. I can't think of a good technical reason not to colocate it with the build cache. We wanted to store interesting inputs in a place the user has access to and somewhere that won't be indexed by desktop searching; that implies a location within the build cache or right next to it.

it is being cleaned by different go clean flags

Cached test results are also stored in the build cache and are also cleaned with a different flag.

it has a different lifetime

The cache eviction code will be able to tell the files apart. Cached fuzz inputs go in a separate subdirectory.

it is a bit more valuable than a build cache (it takes more time to find interesting inputs than "just" compilation);

We're considering a longer cache lifetime because of this. The linked issue above suggests removing fuzz inputs after 30 days without use. We do want some automatic eviction. If a fuzz target is removed or its input types are changed incompatibly, there's no reason to keep the cached files around. It's always possible to copy these files into testdata to preserve them permanently and share them with other developers.

@mvdan
Copy link
Member

mvdan commented Oct 11, 2021

more valuable than a build cache

For what it's worth, this is the main reason I follow this feature request. If I spend a full hour burning CPU to get a nice corpus, I don't want that to just vanish because I haven't worked on the project for a few weeks.

I guess the answer should be "manually make a copy of the relevant corpus" whether GOFUZZCACHE exists or not. What I mainly want is an easy way to make a copy of the corpuses (corpora?) for a module, and I imagined that go env GOFUZZCACHE would be much saner than assuming $(go env GOCACHE)/fuzz.

@AlekSi
Copy link
Contributor Author

AlekSi commented Oct 11, 2021

that implies a location within the build cache or right next to it

I totally agree with "right next to it". But then that location should be configurable.

Cached test results are also stored in the build cache and are also cleaned with a different flag.

Well, that's not entirely true: go clean -cache removes both the build cache and the test cache; go clean -fuzzcache removes neither. From that behavior, it is clear that the build cache includes the test cache but not the fuzz cache. That's why I propose to separate them on the file system too.

We didn't feel like it added enough value to justify the extra complexity in the CLI.

Those environment variables are for advanced use-cases and users only. For example, I want to cache those caches separately on the CI. And you already expose this complexity via go clean flags as shown above; adding this environment variable adds some code to maintain, but not complexity for users.

@toothrot toothrot added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Oct 12, 2021
@toothrot toothrot added this to the Backlog milestone Oct 12, 2021
@katiehockman katiehockman modified the milestones: Backlog, Go1.19 Oct 12, 2021
@katiehockman
Copy link
Contributor

This isn't something that we can dedicate time for in 1.18, but let's re-visit for 1.19. I'm hopeful that 1.18 will provide some good user experience stories and help us better understand which features are most important. The extra context you provided here will be helpful it making that decision too.

@ianlancetaylor
Copy link
Contributor

CC @golang/fuzzing

Moving to Backlog. Please recategorize as appropriate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FeatureRequest fuzz Issues related to native fuzzing support NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
Status: No status
Status: No status
Development

No branches or pull requests

7 participants