Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: test -fuzz seems to print a few "FAIL" lines when stopped via SIGINT #48633

Closed
mvdan opened this issue Sep 26, 2021 · 6 comments
Closed
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone

Comments

@mvdan
Copy link
Member

mvdan commented Sep 26, 2021

For example, with https://github.com/mvdan/sh as a sample module that has a fuzz function:

$ git clone https://github.com/mvdan/sh
$ cd sh
$ git checkout 3d62b69e2043d02c69d7d0efaef371cf9dfcded0
$ cd syntax
$ go version
go version devel go1.18-aeea5bacbf Sat Sep 25 17:12:41 2021 +0000 linux/amd64
$ go test -run=- -vet=off -fuzz=Quote
fuzz: elapsed: 0s, gathering baseline coverage: 0/142 completed
fuzz: elapsed: 0s, gathering baseline coverage: 142/142 completed, now fuzzing with 16 workers
fuzz: elapsed: 3s, execs: 899119 (299642/sec), interesting: 0
fuzz: elapsed: 6s, execs: 1772118 (295318/sec), interesting: 0
^CFAIL
FAIL
FAIL
FAIL
fuzz: elapsed: 7s, execs: 1971588 (293890/sec), interesting: 0
PASS
ok  	mvdan.cc/sh/v3/syntax	6.728s

I imagine that the FAIL ourput lines shouldn't happen at all; nothing has failed, I've just stopped the fuzzing after a few seconds. The final line even says "ok" :)

cc @katiehockman @jayconrod

@jayconrod jayconrod added fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done. release-blocker labels Sep 27, 2021
@jayconrod jayconrod added this to the Go1.18 milestone Sep 27, 2021
@gopherbot
Copy link

Change https://golang.org/cl/352892 mentions this issue: internal/fuzz: disconnect stdout and stderr from the worker

@mvdan
Copy link
Member Author

mvdan commented Sep 28, 2021

Thank you for the quick turnaround!

@katiehockman katiehockman self-assigned this Sep 28, 2021
@katiehockman
Copy link
Contributor

@mvdan can you clarify whether this is happening every time you run this, or if it's just a flaky issue that you run into? My suspicion is that there's just some extraneous FAIL message coming from a worker that was interrupted, which is what you're seeing, but it's hard to be sure.

@mvdan
Copy link
Member Author

mvdan commented Sep 28, 2021

The number of FAIL lines varies each time. I have 16 logical CPU cores, so the fuzzer launches 16 workers. Sometimes I see just a few FAIL lines, some other times I see over a dozen.

I don't think I've ever seen zero FAIL lines after a ^C. I just tried six times in a row, and it always showed at least a handful each time. So it's definitely not a rare occurrence for me.

@katiehockman
Copy link
Contributor

I was just able to reproduce this locally even with the change that I mailed. So there is something else going on. (At best, my change will simply decrease the number of "FAIL" lines that occur, which doesn't really fix the issue)

@katiehockman
Copy link
Contributor

katiehockman commented Sep 29, 2021

I am still not sure how we are going to fix this, but I've at least narrowed it down to https://github.com/golang/go/blob/master/src/cmd/go/internal/test/test.go#L1797. When I run this locally with my patch, this is the FAIL line that is being printed (i.e. the one coming from the main go command process). Will look into how to avoid this.

Edit: the merged CL should fix the issue. I went ahead and disabled printExitStatus when -fuzz is provided.

@golang golang locked and limited conversation to collaborators Jun 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
Status: No status
Development

No branches or pull requests

4 participants