Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: hangs in "found a crash, minimizing..." #46634

Closed
klauspost opened this issue Jun 7, 2021 · 5 comments
Closed

cmd/go: hangs in "found a crash, minimizing..." #46634

klauspost opened this issue Jun 7, 2021 · 5 comments
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Go1.18

Comments

@klauspost
Copy link
Contributor

klauspost commented Jun 7, 2021 •
edited

What version of Go are you using (go version)?

λ gotip version
go version devel go1.17-5542c10fbf Fri Jun 4 03:07:33 2021 +0000 windows/amd64

What operating system and processor architecture are you using (go env)?

go env Output
λ gotip env                                                                                                                                  
set GO111MODULE=                                                                                                                             
set GOARCH=amd64                                                                                                                             
set GOBIN=                                                                                                                                   
set GOCACHE=d:\temp\gocache                                                                                                                  
set GOENV=C:\Users\klaus\AppData\Roaming\go\env                                                                                              
set GOEXE=.exe                                                                                                                               
set GOFLAGS=                                                                                                                                 
set GOHOSTARCH=amd64                                                                                                                         
set GOHOSTOS=windows                                                                                                                         
set GOINSECURE=                                                                                                                              
set GOMODCACHE=e:\gopath\pkg\mod                                                                                                             
set GONOPROXY=                                                                                                                               
set GONOSUMDB=                                                                                                                               
set GOOS=windows                                                                                                                             
set GOPATH=e:\gopath                                                                                                                         
set GOPRIVATE=                                                                                                                               
set GOPROXY=https://goproxy.io                                                                                                               
set GOROOT=C:\Users\klaus\sdk\gotip                                                                                                          
set GOSUMDB=sum.golang.org                                                                                                                   
set GOTMPDIR=                                                                                                                                
set GOTOOLDIR=C:\Users\klaus\sdk\gotip\pkg\tool\windows_amd64                                                                                
set GOVCS=                                                                                                                                   
set GOVERSION=devel go1.17-5542c10fbf Fri Jun 4 03:07:33 2021 +0000                                                                          
set GCCGO=gccgo                                                                                                                              
set AR=ar                                                                                                                                    
set CC=gcc                                                                                                                                   
set CXX=g++                                                                                                                                  
set CGO_ENABLED=1                                                                                                                            
set GOMOD=NUL                                                                                                                                
set CGO_CFLAGS=-g -O2                                                                                                                        
set CGO_CPPFLAGS=                                                                                                                            
set CGO_CXXFLAGS=-g -O2                                                                                                                      
set CGO_FFLAGS=-g -O2                                                                                                                        
set CGO_LDFLAGS=-g -O2                                                                                                                       
set PKG_CONFIG=pkg-config                                                                                                                    
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=d:\temp\wintemp\go-build1674875180=/tmp/go-build -gno-record-gcc-switches

What did you do?

Ran a fuzz test, that ended up eating up all memory, causing OOMs to happen. This is Windows, so applications just crash and aren't killed by the OS.

Likely one or more of the workers crashed.

What did you expect to see?

The controller printed found a crash, minimizing..., but never progressed from there.

Left it for about 15 minutes before I killed it.

What did you see instead?

Fuzzer not getting stuck.
@katiehockman katiehockman changed the title [dev.fuzz] Hangs in "found a crash, minimizing..." [dev.fuzz] cmd/go: hangs in "found a crash, minimizing..." Jun 7, 2021
@katiehockman katiehockman added fuzz Issues related to native fuzzing support NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 7, 2021
@katiehockman katiehockman added this to the Unreleased milestone Jun 7, 2021
@rolandshoemaker
Copy link
Member

This is a complicated problem, since Go doesn't provide any real way to deal with GC backpressure (see #29696 and the associated issues for background). Unless the worker process actually dies, killing the pipe used for RPC communication, the worker/coordinator has no way of knowing that there is a memory problem.

A simple-ish solution would be to just add a timeout to some/all of the RPC calls, so we can just bail out if the worker is taking longer than expected (and we assume it is dead). One question is, if the RPC times out, do we consider that a crash? (i.e. if a fuzz call times out, we copy the input and consider that a crasher).

Once we allow for continuous fuzzing (continuing when a crasher is find, rather than immediately exiting), we would need a way of marking a worker process dead, and ideally restarting it so we don't slowly run out of workers.

@katiehockman
Copy link
Contributor

This is likely going to be a release blocker for Go 1.18.

@katiehockman katiehockman modified the milestones: Unreleased, Backlog Sep 14, 2021
@katiehockman katiehockman modified the milestones: Backlog, Go1.18 Sep 14, 2021
@rsc rsc changed the title [dev.fuzz] cmd/go: hangs in "found a crash, minimizing..." cmd/go: hangs in "found a crash, minimizing..." Sep 21, 2021
@toothrot
Copy link
Contributor

@katiehockman @rolandshoemaker Friendly ping that this issue is marked as a release-blocker for Go 1.18.

@katiehockman katiehockman self-assigned this Oct 14, 2021
@katiehockman
Copy link
Contributor

@klauspost are you still able to reproduce this? We've done a lot of fixes with minimization and timeouts, and I am not able to reproduce this locally anymore.
If you are able to reproduce it, could you provide the fuzz target you are using?

@katiehockman katiehockman added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Oct 28, 2021
@katiehockman
Copy link
Contributor

I'm going to go ahead and close this. Please file a new issue if you experience this issue again.

@golang golang locked and limited conversation to collaborators Nov 10, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Go Security
Status: No status
Milestone
Go1.18
Development

No branches or pull requests
5 participants
@toothrot @rolandshoemaker @klauspost @gopherbot @katiehockman