Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: minimization takes longer than expected for very large []byte/string #48820

Open
katiehockman opened this issue Oct 6, 2021 · 5 comments
Labels
fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@katiehockman
Copy link
Contributor

katiehockman commented Oct 6, 2021

To reproduce, fuzz the following target with go test -fuzz Fuzz

func FuzzFoo(f *testing.F) {
	f.Fuzz(func(t *testing.T, b []byte) {
		if len(b) > 10000 {
			panic("bad")
		}
	})
}

The following output will occur:

fuzz: elapsed: 0s, gathering baseline coverage: 0/1 completed
fuzz: elapsed: 0s, gathering baseline coverage: 1/1 completed, now fuzzing with 8 workers
fuzz: minimizing 34995-byte crash input...
fuzz: elapsed: 3s, execs: 13821 (4604/sec), new interesting: 0 (total: 1)
fuzz: elapsed: 6s, execs: 13821 (2303/sec), new interesting: 0 (total: 1)
fuzz: elapsed: 9s, execs: 13821 (1535/sec), new interesting: 0 (total: 1)

Notice that it found a crash and indicated that it would minimize it, but then just proceeded with fuzzing anyway (Correction: It didn't keep fuzzing, it just kept updating the log. It looks like the worker hung). If you press Ctrl^C, it exits and writes the crasher:

fuzz: elapsed: 0s, gathering baseline coverage: 0/1 completed
fuzz: elapsed: 0s, gathering baseline coverage: 1/1 completed, now fuzzing with 8 workers
fuzz: minimizing 34995-byte crash input...
fuzz: elapsed: 3s, execs: 13821 (4604/sec), new interesting: 0 (total: 1)
fuzz: elapsed: 6s, execs: 13821 (2303/sec), new interesting: 0 (total: 1)
fuzz: elapsed: 9s, execs: 13821 (1535/sec), new interesting: 0 (total: 1)
fuzz: elapsed: 12s, execs: 13821 (1152/sec), new interesting: 0 (total: 1)
^Cfuzz: elapsed: 14s, execs: 13821 (994/sec), new interesting: 0 (total: 1)
--- FAIL: FuzzAdd (13.90s)
        --- FAIL: FuzzAdd (0.00s)
            testing.go:1299: panic: bad
                goroutine 1506 [running]:
                runtime/debug.Stack()
                        /Users/katiehockman/godev/src/runtime/debug/stack.go:24 +0x90
                testing.tRunner.func1()
                        /Users/katiehockman/godev/src/testing/testing.go:1299 +0x5a5
                ...
                
    
    Crash written to testdata/fuzz/FuzzAdd/618b91bbfb35dd0736e9c6c90ed38973a2e1f5ebb7edf50714c31ae21b18d419
    To re-run:
    go test foo -run=FuzzAdd/618b91bbfb35dd0736e9c6c90ed38973a2e1f5ebb7edf50714c31ae21b18d419
FAIL
exit status 1
FAIL    foo     18.072s

Turning off minimization fixes this (e.g. go test -fuzz Fuzz -fuzzminimizetime=0x)

This also only seems to happen once the fuzzing engine has already run baseline coverage or found a new interesting value. If the target is changed to panic for every input, this bug won't occur and it'll crash as expected. e.g.

func FuzzAdd(f *testing.F) {
	f.Fuzz(func(t *testing.T, b []byte) {
		panic("bad")
	})
}

This also shows a hole in our testing. Most of our tests use -fuzztime in order to ensure they don't run for too long. However, using -fuzztime ends fuzzing after a certain number of iterations. That means we aren't testing that the fuzzing process will complete successfully if a crash occurs without help from -fuzztime.

/cc @golang/fuzzing

@katiehockman katiehockman added NeedsFix The path to resolution is known, but the work has not been done. release-blocker fuzz Issues related to native fuzzing support labels Oct 6, 2021
@katiehockman katiehockman added this to the Go1.18 milestone Oct 6, 2021
@katiehockman
Copy link
Contributor Author

katiehockman commented Oct 6, 2021

I figured out what's happening. For the example I provided, the crash only occurs if the []byte is at least 10,000 bytes. The original crash is 34,995 bytes long, which takes a long time to minimize. The first two minimization efforts (cutting the tail and removing every byte) run pretty quickly. However, removing every subset of bytes takes a very long time if the input is long.

If I am patient and just let it run for a while, then it will eventually hit that 1 minute minimization timeout and stop gracefully. The part that I found confusing is that it continues to print these "fuzz: elapsed:" messages after minimization has begun, which is a bit jarring and makes it look like the process is hanging. We should probably either 1) stop printing these log messages when we're minimizing or 2) update them to indicate the minimization progress. For (2) that could mean something like "fuzz: elapsed: 12s, minimizing"

@rsc
Copy link
Contributor

rsc commented Oct 6, 2021

(2) is much better, so the user knows it's not hung. If you were confused, they will be confused.

We should also probably do something about the removing subsets of bytes taking forever, but let's handle 'taking longer than user expected' gracefully first.

@katiehockman
Copy link
Contributor Author

Yep agreed. I have a patch ready, and will send that shortly.

@katiehockman katiehockman changed the title testing: fuzzing is getting stuck in minimization testing: minimization takes longer than expected for very large []byte/string Oct 6, 2021
@gopherbot
Copy link

Change https://golang.org/cl/354371 mentions this issue: internal/fuzz: log that minimization is occurring

gopherbot pushed a commit that referenced this issue Oct 7, 2021
Previously, when fuzzing for a period of time, the
command line output would look something like this:

   fuzz: minimizing 34995-byte crash input...
   fuzz: elapsed: 3s, execs: 13821 (4604/sec), new interesting: 0 (total: 1)
   fuzz: elapsed: 6s, execs: 13821 (2303/sec), new interesting: 0 (total: 1)
   fuzz: elapsed: 9s, execs: 13821 (1535/sec), new interesting: 0 (total: 1)
   --- FAIL: FuzzFoo (9.05s)

This is the same output it has while fuzzing, so if
minimization runs for a long time (default allows 1
minute), then it looks like minimization is hanging.
It's also confusing that the execs/sec would continually
decrease.

Now, when minimization is running, the command line
output will look something like this:

   fuzz: minimizing 34995-byte crash input...
   fuzz: elapsed: 3s, minimizing
   fuzz: elapsed: 6s, minimizing
   fuzz: elapsed: 9s, minimizing
   fuzz: elapsed: 9s, minimizing
   --- FAIL: FuzzFoo (9.05s)

The final "fuzz: elapsed: 6s, minimizing" could be
printed twice because we always print one final log
to the command line before we exit.

Updates #48820

Change-Id: Ie5b9fde48b8d4e36e13a81ae50a6d69bf4d0dbe3
Reviewed-on: https://go-review.googlesource.com/c/go/+/354371
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@katiehockman katiehockman modified the milestones: Go1.18, Backlog Nov 10, 2021
@katiehockman
Copy link
Contributor Author

Moving this to the backlog for now, since at least (2) is done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done.
Projects
Status: No status
Development

No branches or pull requests

3 participants