runtime: SIGPROF during copystack causing process deadlock on single core arm? #44791
Labels
FrozenDueToAge
NeedsInvestigation
Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
WaitingForInfo
Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Comments
jdizzle
changed the title
ianlancetaylor
changed the title
ianlancetaylor
added
the
NeedsInvestigation
Could you try a newer version of Go, maybe Go 1.16? There have been a few changes with profiling, signal handling, etc. And Go 1.13 is no longer supported. Thanks. |
cherrymui
added
the
WaitingForInfo
|
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Unknown
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
pprof.StartCPUProfile() while other go-routines are running. Some cgo calls, some http connections.
What did you expect to see?
No unusual behavior while profiling is enabled.
What did you see instead?
System deadlocks. All go routines make no process. All but one thread is idle. The non-idle thread consumes 100% cpu.
If I pull the stack pointer from the linux proc pid task stat file:
/proc/1154/task/1162 # kill -SIGSTOP 1154 /proc/1154/task/1162 # awk '{printf("sp: 0x%x, pc: 0x%x\n", $29, $30)}' stat sp: 0x280b864, pc: 0x87228And look up the PC, it appears to be working through a stack trace of some kind.
Examining the proc pid task status files I see that all signals are blocked on this thread as well:
If I examine this same process in another instance that is not having this problem, the only difference is theres no thread with the
fffffffffffbfeffmask.I have dumped the stack at the sp reported by the stat file by reading the
memfile. Apparently unwinding the stack requires per-function stack-usage info encoded into the binary that I'm not sure how to get. If I just look for related PC values I can piece together a call stack in the SIGPROF signal handler call tree.and you can see that 00f3 0000001b is the arg of SIGPROF signal value to runtime.sigtramp.
If I SIGABRT the process it will exit and produce a call trace on stderr. A consistent artifact we see in the aborted call trace of a stuck process is a go routine stuck in a "copystack" phase:
goroutine 3722564 [copystack]: sync.(*Once).Do(0x229051c, 0x26b6be8) /code/go/src/sync/once.go:40 +0x50 net/http.(*Transport).roundTrip(0x2290478, 0x26bf800, 0x0, 0x0, 0x0) /code/go/src/net/http/transport.go:455 +0x4c net/http.(*Transport).RoundTrip(0x2290478, 0x26bf800, 0x2290478, 0x0, 0x0) /code/go/src/net/http/roundtrip.go:17 +0x24 net/http.send(0x26bf800, 0x168f470, 0x2290478, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2639b00, 0x100, ...) /code/go/src/net/http/client.go:250 +0x37c net/http.(*Client).send(0x232d230, 0x26bf800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2, 0x25d4420, 0x20, ...) /code/go/src/net/http/client.go:174 +0xb8 net/http.(*Client).do(0x232d230, 0x26bf800, 0x0, 0x0, 0x0) /code/go/src/net/http/client.go:641 +0x2c8 net/http.(*Client).Do(...) /code/go/src/net/http/client.go:509 net/http.(*Client).Post(0x232d230, 0x13fa024, 0x20, 0x13d346d, 0x14, 0x168e750, 0x25d43a0, 0x2693b80, 0x0, 0x6bc78) /code/go/src/net/http/client.go:769 +0x134 net/http.Post(...) /code/go/src/net/http/client.go:749 firmware.samsaradev.io/logevent.doLogPushHttp(0x13fa024, 0x20, 0x24bc280, 0x96, 0x96, 0x2693bc0) /code/samsara/go/src/firmware.samsaradev.io/logevent/manager.go:767 +0xd4 created by firmware.samsaradev.io/logevent.(*loggerManager).maybeSendLogs /code/samsara/go/src/firmware.samsaradev.io/logevent/manager.go:761 +0x66cI can see some evidence of this being the callstack being decoded in the pprof signal stack as well. Here's a contiguous set of values on the stack that happens to resemble the call stack dumped by SIGABRT very closely:
The text was updated successfully, but these errors were encountered: