cmd/go: simple off-by-one example takes longer than expected to complete #47090
Comments
mknyszek
added
the
NeedsInvestigation
|
CC @golang/fuzzing |
|
This is an interesting comparison since the target expects a very specific input size. It seems plausible that go-fuzz is much more conservative about increasing the size of inputs during mutation (perhaps in relation to the initial size of the input), so when starting from an empty input you're more likely to hit an input which is 9 bytes long, whereas our mutators don't really take this into account, making it much more of a lottery. |
|
I have a similar example here: https://github.com/stevenjohnstone/go-beta-fuzzer-vs-libfuzzer/blob/main/fuzz.go. In this case, the number of bytes a mutator needs to "guess" is fewer so it completes in a reasonable time. In that repo, I build a libfuzzer harness equivalent to a native fuzzer. Both use the "libfuzzer" build for instrumentation (inline 8-bit counters and cmp tracing hooks) so they have the same feedback (modulo some noise from supporting code) for executions of the function under test. I disable the cmp tracing functionality during libfuzzer runs to make sure I'm comparing apples with apples. Initial experiments seem to indicate that libfuzzer is able to find a crasher about 100x faster than the native fuzzer for my very artificial example. I don't think this can be used as judgement on effectiveness without a lot of careful thought but I think it's interesting that we have an alternative implementation available for comparisons. |
rsc
changed the title
|
Coming back to look at this again, the example case is a great example of something fuzzers are generally not great at without additional instrumentation strategies. In order to hit the out-of-bounds access, the fuzzer first has to generate a very particular string ( I think we should probably just close this, in favor of #46507 which details more advanced strategies the fuzzer could incorporate in the future. |
|
@rolandshoemaker I don't think this needs any instrumentation of comparisons to work. If I build the example as package fuzz
func FuzzLibFuzzer(data []byte) int {
if len(data) == 9 {
if data[0] == 'G' && data[1] == 'O' && data[2] == 'P' && data[3] == 'H' && data[4] == 'E' && data[5] == 'R' && data[6] == 'S' && data[7] == '!' && data[8] == '!' && data[9] == '!' {
panic("found")
}
}
return 0
}and compile it for use with libfuzzer (with https://github.com/stevenjohnstone/go114-fuzz-build and gotip as the go compiler to make the comparison fair), then the fuzzer finds the crasher in a few seconds even with comparison tracing disabled: I think you hit the nail on the head in #47090 (comment): the fuzzer tries really long inputs. When I instrument with bpftrace and run the golang native fuzzer for 30 seconds, I see |
|
Oh hm, now looking at the mutation code for the ~third time, I'm seeing some somewhat unexpected behavior. With a handful of architectural tweaks I can mostly match libFuzzer performance on this particular target. I need to stare at this a little bit longer to figure out the correct solution. |
|
Change https://golang.org/cl/364214 mentions this issue: |
dmitshur
added
NeedsFix
What version of Go are you using (
go version)?What did you do?
I ported a simple example I've used to test
go-fuzzin the past:⬇️
$ gotip test -fuzz=FuzzOffByOneWhat did you expect to see?
I've run the example several times before, and would generally expect a fuzzer to find a crash fairly fast, even without a corpus -- somewhere between a few seconds to a few minutes.
Using the same CPU and RAM configuration as the native fuzzer test, I was able to find it within seconds using
go-fuzz:What did you see instead?
With the new native fuzzer, it took ~24 hours to find a crash using 2GB of RAM and 1 CPU (from an
n1-standard-2instance):The text was updated successfully, but these errors were encountered: