Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testing: flaky issue where crash written doesn't reproduce an error #48926

Closed
katiehockman opened this issue Oct 12, 2021 · 3 comments
Closed

testing: flaky issue where crash written doesn't reproduce an error #48926

katiehockman opened this issue Oct 12, 2021 · 3 comments
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone
Go1.18

Comments

@katiehockman
Copy link
Contributor

I've been playing around with this fuzz test:

func FuzzFuzzer(f *testing.F) {
	f.Fuzz(func(t *testing.T, x int, s string) {
		b := []byte(s)
		if bytes.Contains(b, strconv.AppendInt(nil, int64(x), 10)) {
			t.Errorf("minimize this! %d, len(s)=%d", x, len(s))
		}
	})
}

I've noticed that sometimes it is writing a crash that doesn't actually reproduce an error when re-run with go test. My guess is that this is a minimization issue, since I haven't been able to reproduce this bug when I set -fuzzminimizetime=0.

Still trying to figure out where the bug was introduced and why none of our tests are catching it.

/cc @golang/fuzzing
@katiehockman katiehockman added NeedsFix The path to resolution is known, but the work has not been done. release-blocker fuzz Issues related to native fuzzing support labels Oct 12, 2021
@katiehockman katiehockman added this to the Go1.18 milestone Oct 12, 2021
@katiehockman
Copy link
Contributor Author

Something very weird is going on. I'm also seeing times where minimization wasn't even attempted. Those crashes are also not reproducible when run with go test

e.g.

godev test -run Fuzz -fuzz Fuzz -v -parallel 5
=== FUZZ  FuzzFuzzer
fuzz: elapsed: 0s, gathering baseline coverage: 0/13 completed
fuzz: elapsed: 0s, gathering baseline coverage: 13/13 completed, now fuzzing with 5 workers
fuzz: elapsed: 0s, execs: 100 (1508/sec), new interesting: 0 (total: 9)
--- FAIL: FuzzFuzzer (0.07s)
        --- FAIL: FuzzFuzzer (0.00s)
            foo_test.go:20: minimize this!
    
    Crash written to testdata/fuzz/FuzzFuzzer/9bbe05c1b66fbfdbd1d075945dfe9b96d1b56a04f8edf276fd6b363a92edfb5e
    To re-run:
    go test foo -run=FuzzFuzzer/9bbe05c1b66fbfdbd1d075945dfe9b96d1b56a04f8edf276fd6b363a92edfb5e
FAIL
exit status 1
FAIL    foo     0.214s

@katiehockman
Copy link
Contributor Author

Figured it out. If a crash occurs while minimizing an interesting input, we don't write that newly crashing input to memory, or set it to the entryOut that is read by the coordinator. I have a patch that fixes it, so I'll clean that up and add a test, and plan to send the fix tomorrow 👍

@katiehockman
Copy link
Contributor Author

Duplicate of #48731

@katiehockman katiehockman marked this as a duplicate of #48731 Oct 13, 2021
@golang golang locked and limited conversation to collaborators Oct 13, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
Go Security
Status: No status
Milestone
Go1.18
Development

No branches or pull requests
2 participants
@gopherbot @katiehockman