Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: visualize fuzz corpus coverage #46765

Open
rolandshoemaker opened this issue Jun 15, 2021 · 6 comments
Open

testing: visualize fuzz corpus coverage #46765

rolandshoemaker opened this issue Jun 15, 2021 · 6 comments
Labels
FeatureRequest fuzz Issues related to native fuzzing support NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@rolandshoemaker
Copy link
Member

It would be nice to be able to visualize the source coverage provided by an input corpus, in a similar manner to the coverage information provided for unit tests.

Fuzzing counters are implemented in a slightly different fashion to the (testing) coverage counters, but probably a similar method can be used to produce compatible coverage profiles for use with go tool cover.

@rolandshoemaker rolandshoemaker added the fuzz Issues related to native fuzzing support label Jun 15, 2021
@rolandshoemaker
Copy link
Member Author

This may be more complicated than it seemed on the surface.

Since we add the libfuzzer-style counters during compilation, while doing other things to the source (like in-lining), the code we are instrumenting does not necessarily look like the source on disk, making mapping from counter positions to source positions complicated.

One other approach would be to just use the existing go tool cover coverage instrumentation, in addition to the libfuzzer instrumentation, since it already exists. There may be downsides I cannot think of at the moment, but it seems like a rather simple approach.

@timothy-king
Copy link
Contributor

One other approach would be to just use the existing go tool cover coverage instrumentation, in addition to the libfuzzer instrumentation, since it already exists. There may be downsides I cannot think of at the moment, but it seems like a rather simple approach.

IIUC the suggestion is to instrument the program with two different coverage metrics, produce a single binary where both coverage metrics are aggregated during execution, but the two different metrics are post processed differently. So go tool cover is used for producing reports and the libfuzzer style counters are used for determining if coverage increased. Is this an accurate understanding of the approach?

Another approach could be to re-run the fuzzer generated tests under a go tool cover instrumented binary to produce the coverage information. FWIW other fuzzing systems have accepted a similar division of labor at the cost of needing to compile 2 binaries.

@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 17, 2021
@cagedmantis cagedmantis added this to the Backlog milestone Jun 17, 2021
@cagedmantis
Copy link
Contributor

/cc @golang/fuzzing

@rsc rsc changed the title [dev.fuzz] testing: visualize corpus coverage testing: visualize fuzz corpus fuzz Sep 21, 2021
@rsc rsc changed the title testing: visualize fuzz corpus fuzz testing: visualize fuzz corpus coverage Sep 21, 2021
@firelizzard18
Copy link
Contributor

Another approach could be to re-run the fuzzer generated tests under a go tool cover instrumented binary to produce the coverage information. FWIW other fuzzing systems have accepted a similar division of labor at the cost of needing to compile 2 binaries.

It seems like this could be done with a minimal change: add a -fuzztestdata <dir> flag to go test -fuzz=FuzzFoo ./pkg that tells the fuzzer to load the generated corpus from <dir> instead of ./pkg/testdata/fuzz. Then users could do it themselves with go test -run=FuzzFoo -fuzztestdata $GOCACHE/fuzz/example.com/module/pkg -cover.

@randall77
Copy link
Contributor

I just ran into wanting this.

% go test -fuzz=MyTest -coverprofile=cov.out .
cannot use -coverprofile flag with -fuzz flag

It would be nice if it were that simple.

I'd settle for some way to output the generated corpus, after a -fuzztime=, so that I could copy it into testdata/fuzz somehow. Then a normal test the next time around, with -coverprofile, would touch what the fuzzer was touching.

@ericchiang
Copy link
Contributor

Apologies if this redundant, but I wanted to look at the fuzzing coverage for a project today and found this issue.

I ran the following commands as a workaround to copy the fuzz cache to "testdata/fuzz" and generate a cover profile:

mkdir -p testdata/fuzz
cp -r "$( go env GOCACHE )/fuzz/$( go list )/" testdata/fuzz
go test -coverprofile=coverage.out
go tool cover -html=coverage.out

which seems to work today

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FeatureRequest fuzz Issues related to native fuzzing support NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Status: No status
Development

No branches or pull requests

7 participants