Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: netpoll throw: break fd ready for something unexpected #67056

Open
chalkytoast opened this issue Apr 26, 2024 · 2 comments
Open

runtime: netpoll throw: break fd ready for something unexpected #67056

chalkytoast opened this issue Apr 26, 2024 · 2 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@chalkytoast
Copy link

Go version

go version go1.21.6 linux/arm

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm'
GOBIN=''
GOCACHE='/home/pi/.cache/go-build'
GOENV='/home/pi/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/pi/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/pi/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/pi/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/pi/go/pkg/tool/linux_arm'
GOVCS=''
GOVERSION='go1.21.6'
GCCGO='gccgo'
GOARM='6'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -marm -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1411701886=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Long running task crashed overnight in vicinity of epoll. No user interaction.

What did you see happen?

We have observed this crash a couple of times over the past year. It is very sporadic and not reproducable. Always on Raspberry Pi. OS version is 11.8.

runtime: netpoll: break fd ready for 5                                                       
fatal error: runtime: netpoll: break fd ready for something unexpected                       
                                                                                             
runtime stack:                                
runtime.throw({0x30936a, 0x39})                                                                                                                                                            
        /home/pi/go/src/runtime/panic.go:1077 +0x4c fp=0xeaffe4b4 sp=0xeaffe4a0 pc=0x4f0c4   
runtime.netpoll(0x0)                                                                         
        /home/pi/go/src/runtime/netpoll_epoll.go:141 +0x3c4 fp=0xeaffed1c sp=0xeaffe4b4 pc=0x4ace0                                                                                         
runtime.findRunnable()                                                                       
        /home/pi/go/src/runtime/proc.go:2978 +0x360 fp=0xeaffedb8 sp=0xeaffed1c pc=0x57ff8   
runtime.schedule()                                                                           
        /home/pi/go/src/runtime/proc.go:3582 +0xac fp=0xeaffedd4 sp=0xeaffedb8 pc=0x59d24    
runtime.park_m(0x883f00)                                                                     
        /home/pi/go/src/runtime/proc.go:3745 +0x160 fp=0xeaffedf0 sp=0xeaffedd4 pc=0x5a570   
traceback: unexpected SPWRITE function runtime.mcall                                         
runtime.mcall()                               
        /home/pi/go/src/runtime/asm_arm.s:265 +0x48 fp=0xeaffedf8 sp=0xeaffedf0 pc=0x84f80

goroutine 1 [IO wait]:                        
runtime.gopark(0x311298, 0xeeb1bee0, 0x2, 0x1b, 0x5)                                         
        /home/pi/go/src/runtime/proc.go:398 +0x104 fp=0x83cd28 sp=0x83cd14 pc=0x52874        
runtime.netpollblock(0xeeb1bed0, 0x72, 0x0)                                                  
        /home/pi/go/src/runtime/netpoll.go:564 +0x100 fp=0x83cd40 sp=0x83cd28 pc=0x49ef8     
internal/poll.runtime_pollWait(0xeeb1bed0, 0x72)                                             
        /home/pi/go/src/runtime/netpoll.go:343 +0x54 fp=0x83cd54 sp=0x83cd40 pc=0x823cc      
internal/poll.(*pollDesc).wait(0x918158, 0x72, 0x0)                                          
        /home/pi/go/src/internal/poll/fd_poll_runtime.go:84 +0x30 fp=0x83cd68 sp=0x83cd54 pc=0xd3608                                                                                       
internal/poll.(*pollDesc).waitRead(...)                                                      
        /home/pi/go/src/internal/poll/fd_poll_runtime.go:89                                  
internal/poll.(*FD).Accept(0x918140)          
        /home/pi/go/src/internal/poll/fd_unix.go:611 +0x2c0 fp=0x83cdb8 sp=0x83cd68 pc=0xd4a08                                                                                             
net.(*netFD).accept(0x918140)                 
        /home/pi/go/src/net/fd_unix.go:172 +0x20 fp=0x83ce18 sp=0x83cdb8 pc=0x1b82b4         
net.(*TCPListener).accept(0x930090)           
        /home/pi/go/src/net/tcpsock_posix.go:152 +0x20 fp=0x83ce30 sp=0x83ce18 pc=0x1ca670   
net.(*TCPListener).Accept(0x930090)           
        /home/pi/go/src/net/tcpsock.go:315 +0x30 fp=0x83ce4c sp=0x83ce30 pc=0x1c980c         
net/http.(*onceCloseListener).Accept(0x914860)                                               
        <autogenerated>:1 +0x34 fp=0x83ce64 sp=0x83ce4c pc=0x27bba0                          
net/http.(*Server).Serve(0x95a000, {0x360df8, 0x930090})                                     
        /home/pi/go/src/net/http/server.go:3056 +0x330 fp=0x83cf00 sp=0x83ce64 pc=0x2704c4   
net/http.(*Server).ListenAndServe(0x95a000)                                                  
        /home/pi/go/src/net/http/server.go:2985 +0xa4 fp=0x83cf24 sp=0x83cf00 pc=0x270144    
net/http.ListenAndServe(...)                  
        /home/pi/go/src/net/http/server.go:3239                                              
main.main()                                   
        /home/pi/foo/cmd/bar/main.go:38 +0x374 fp=0x83cfa0 sp=0x83cf24 pc=0x2907c8                                                                                      
runtime.main()                                
        /home/pi/go/src/runtime/proc.go:267 +0x338 fp=0x83cfec sp=0x83cfa0 pc=0x522dc        
runtime.goexit()                              
        /home/pi/go/src/runtime/asm_arm.s:859 +0x4 fp=0x83cfec sp=0x83cfec pc=0x86c4c

What did you expect to see?

No panic.
@joedian joedian added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 26, 2024
@ianlancetaylor ianlancetaylor changed the title fatal error: runtime: netpoll: break fd ready for something unexpected runtime: netpoll throw: break fd ready for something unexpected Apr 26, 2024
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Apr 26, 2024
@ianlancetaylor
Copy link
Contributor

"ready for 5" means that both EPOLLIN and EPOLLOUT are set, but we only expect EPOLLIN.

I'm not sure what is happening but this was probably fixed for the future Go 1.23 release as a side effect of https://go.dev/cl/560615.

@chalkytoast
Copy link
Author

chalkytoast commented Apr 29, 2024 •
edited

Thanks for looking into this. We'll report when 1.23 is released and had time to evaluate.

@mknyszek mknyszek added this to the Backlog milestone May 8, 2024
@mknyszek mknyszek added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Go Compiler / Runtime
Status: Todo
Milestone
Backlog
Development

No branches or pull requests
5 participants
@mknyszek @ianlancetaylor @gopherbot @joedian @chalkytoast