runtime/pprof: panic in appendLocsForStack on linux-arm64-packet #35538
Labels
Milestone
Comments
bcmills
added
the
NeedsInvestigation
|
Using gomote, printed out the the cached location pcs and the pcs in the stack. It's flaky - all failures I got are only from TestGoroutineSwitch so far. @randall77 === RUN TestGoroutineSwitch
XXX
Cached PCs: [300657 300665 300656 300665 300656 300665]
[{300656 runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}]
[{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}]
[{300655 runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}]
[{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}]
[{300655 runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}]
[{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}]
vs New PCs: [300657 300665 300656 300665]
[{300656 runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}]
[{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}]
[{300655 runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}]
[{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}]
cached loc = 19
panic: runtime error: slice bounds out of range [6:4]
goroutine 251 [running]:
runtime/pprof.(*profileBuilder).appendLocsForStack(0x40001eef20, 0x4000454d00, 0x0, 0x8, 0x4041192478, 0x4, 0x371, 0x0, 0x6, 0x8)
/workdir/go/src/runtime/pprof/proto.go:410 +0x8ac
runtime/pprof.(*profileBuilder).build(0x40001eef20)
/workdir/go/src/runtime/pprof/proto.go:362 +0xfc
runtime/pprof.profileWriter(0x24be00, 0x404118aea0)
/workdir/go/src/runtime/pprof/pprof.go:779 +0xd8
created by runtime/pprof.StartCPUProfile
/workdir/go/src/runtime/pprof/pprof.go:750 +0x128
FAIL runtime/pprof 31.955s
|
|
BTW, I just got an internal report on the same panic, but with linux-amd64, and with the heap profile. |
|
Change https://golang.org/cl/206958 mentions this issue: |
This was referenced Nov 13, 2019
gopherbot
pushed a commit
that referenced
this issue
Nov 14, 2019
tryAdd shouldn't succeed (and accept the new frame) if the last
existing frame on the deck is not an inlined frame.
For example, when we see the followig stack
[300656 300664 300655 300664]
with each PC corresponds to
[{PC:300656 Func:nil Function:runtime.nanotime File:/workdir/go/src/runtime/time_nofake.go Line:19 Entry:300416 {0x28dac8 0x386c80}}]
[{PC:300664 Func:0x28dac8 Function:runtime.checkTimers File:/workdir/go/src/runtime/proc.go Line:2623 Entry:300416 {0x28dac8 0x386c80}}]
[{PC:300655 Func:nil Function:runtime.nanotime File:/workdir/go/src/runtime/time_nofake.go Line:19 Entry:300416 {0x28dac8 0x386c80}}]
[{PC:300664 Func:0x28dac8 Function:runtime.checkTimers File:/workdir/go/src/runtime/proc.go Line:2623 Entry:300416 {0x28dac8 0x386c80}}]
PC:300656 and PC:300664 belong to a single location entry,
but the bug in the current tryAdd logic placed the entire stack into one
location entry.
Also adds tests - this crash is a tricky case to test because I think it
should happen with normal go code. The new TestTryAdd simulates it by
using fake call sequences. The test crashed without the fix.
Update #35538
Change-Id: I6d3483f757abf4c429ab91616e4def90832fc04a
Reviewed-on: https://go-review.googlesource.com/c/go/+/206958
Reviewed-by: Keith Randall <khr@golang.org>
|
Change https://golang.org/cl/207609 mentions this issue: |
hyangah
removed
the
NeedsInvestigation
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2019-11-12T05:24:00-d22b573/linux-arm64-packet
2019-11-12T01:09:31-0ee22d9/linux-arm64-packet
2019-11-08T18:00:31-c444ec3/linux-arm64-packet
CC @hyangah @randall77
The text was updated successfully, but these errors were encountered: