-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime/pprof: panic in appendLocsForStack on linux-arm64-packet #35538
Comments
Using gomote, printed out the the cached location pcs and the pcs in the stack. It's flaky - all failures I got are only from TestGoroutineSwitch so far. @randall77 === RUN TestGoroutineSwitch XXX Cached PCs: [300657 300665 300656 300665 300656 300665] [{300656 runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}] [{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}] [{300655 runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}] [{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}] [{300655 runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}] [{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}] vs New PCs: [300657 300665 300656 300665] [{300656 runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}] [{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}] [{300655 runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}] [{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}] cached loc = 19 panic: runtime error: slice bounds out of range [6:4] goroutine 251 [running]: runtime/pprof.(*profileBuilder).appendLocsForStack(0x40001eef20, 0x4000454d00, 0x0, 0x8, 0x4041192478, 0x4, 0x371, 0x0, 0x6, 0x8) /workdir/go/src/runtime/pprof/proto.go:410 +0x8ac runtime/pprof.(*profileBuilder).build(0x40001eef20) /workdir/go/src/runtime/pprof/proto.go:362 +0xfc runtime/pprof.profileWriter(0x24be00, 0x404118aea0) /workdir/go/src/runtime/pprof/pprof.go:779 +0xd8 created by runtime/pprof.StartCPUProfile /workdir/go/src/runtime/pprof/pprof.go:750 +0x128 FAIL runtime/pprof 31.955s |
BTW, I just got an internal report on the same panic, but with linux-amd64, and with the heap profile. |
Change https://golang.org/cl/206958 mentions this issue: |
tryAdd shouldn't succeed (and accept the new frame) if the last existing frame on the deck is not an inlined frame. For example, when we see the followig stack [300656 300664 300655 300664] with each PC corresponds to [{PC:300656 Func:nil Function:runtime.nanotime File:/workdir/go/src/runtime/time_nofake.go Line:19 Entry:300416 {0x28dac8 0x386c80}}] [{PC:300664 Func:0x28dac8 Function:runtime.checkTimers File:/workdir/go/src/runtime/proc.go Line:2623 Entry:300416 {0x28dac8 0x386c80}}] [{PC:300655 Func:nil Function:runtime.nanotime File:/workdir/go/src/runtime/time_nofake.go Line:19 Entry:300416 {0x28dac8 0x386c80}}] [{PC:300664 Func:0x28dac8 Function:runtime.checkTimers File:/workdir/go/src/runtime/proc.go Line:2623 Entry:300416 {0x28dac8 0x386c80}}] PC:300656 and PC:300664 belong to a single location entry, but the bug in the current tryAdd logic placed the entire stack into one location entry. Also adds tests - this crash is a tricky case to test because I think it should happen with normal go code. The new TestTryAdd simulates it by using fake call sequences. The test crashed without the fix. Update #35538 Change-Id: I6d3483f757abf4c429ab91616e4def90832fc04a Reviewed-on: https://go-review.googlesource.com/c/go/+/206958 Reviewed-by: Keith Randall <khr@golang.org>
Change https://golang.org/cl/207609 mentions this issue: |
2019-11-12T05:24:00-d22b573/linux-arm64-packet
2019-11-12T01:09:31-0ee22d9/linux-arm64-packet
2019-11-08T18:00:31-c444ec3/linux-arm64-packet
CC @hyangah @randall77
The text was updated successfully, but these errors were encountered: