runtime: deadlock during CPU profiling #8407
Labels
Milestone
Comments
CL https://golang.org/cl/112640043 mentions this issue. |
|
This issue was closed by revision e91704a. Status changed to Fixed. |
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jun 25, 2018
This is bad for 2 reasons: 1. if the code under lock ever grows stack, it will deadlock as stack growing acquires mheap lock. 2. It currently deadlocks with SetCPUProfileRate: scavenger locks mheap, receives prof signal and tries to lock prof lock; meanwhile SetCPUProfileRate locks prof lock and tries to grow stack (presumably in runtime.unlock->futexwakeup). Boom. Let's assume that it Fixes golang#8407. LGTM=rsc R=golang-codereviews, rsc CC=golang-codereviews, khr https://golang.org/cl/112640043
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jul 9, 2018
This is bad for 2 reasons: 1. if the code under lock ever grows stack, it will deadlock as stack growing acquires mheap lock. 2. It currently deadlocks with SetCPUProfileRate: scavenger locks mheap, receives prof signal and tries to lock prof lock; meanwhile SetCPUProfileRate locks prof lock and tries to grow stack (presumably in runtime.unlock->futexwakeup). Boom. Let's assume that it Fixes golang#8407. LGTM=rsc R=golang-codereviews, rsc CC=golang-codereviews, khr https://golang.org/cl/112640043
This issue was closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Just observed the following hang on linux/amd64 on tip. Conditions are unclear. I can't reproduce it again. Note there are 2 "running" goroutines at the bottom. SIGABRT: abort PC=0x42e271 goroutine 18 [syscall]: runtime.notetsleepg(0x7f8fa6d19008, 0xffffffffffffffff) src/pkg/runtime/lock_futex.c:198 +0x4d fp=0xc2080afed0 sp=0xc2080afea0 getprofile(0xc2080aff38, 0x7f8fa6d19000) src/pkg/runtime/cpuprof.goc:361 +0x378 fp=0xc2080aff20 sp=0xc2080afed0 runtime.CPUProfile(0x0, 0x0, 0x0) src/pkg/runtime/cpuprof.goc:432 +0x4f fp=0xc2080aff38 sp=0xc2080aff20 runtime/pprof.profileWriter(0x7f8fa71cfcf0, 0xc2080e4000) src/pkg/runtime/pprof/pprof.go:600 +0x1e fp=0xc2080aff98 sp=0xc2080aff38 runtime.goexit() src/pkg/runtime/proc.c:1476 fp=0xc2080affa0 sp=0xc2080aff98 created by runtime/pprof.StartCPUProfile src/pkg/runtime/pprof/pprof.go:594 +0x18c goroutine 1 [chan receive]: runtime.park(0x417dc0, 0xc2080461d8, 0x63e3ec) src/pkg/runtime/proc.c:1407 +0x97 fp=0xc208019c28 sp=0xc208019c18 runtime.parkunlock(0xc2080461d8, 0x63e3ec) src/pkg/runtime/proc.c:1423 +0x3b fp=0xc208019c48 sp=0xc208019c28 chanrecv(0x4fd3a0, 0xc208046180, 0xc208019df0, 0x442001, 0x0) src/pkg/runtime/chan.goc:301 +0x4e4 fp=0xc208019ca8 sp=0xc208019c48 runtime.chanrecv1(0x4fd3a0, 0xc208046180, 0xc208019df0) src/pkg/runtime/chan.goc:385 +0x38 fp=0xc208019cd8 sp=0xc208019ca8 testing.RunTests(0x5a89e8, 0x63cd80, 0x6, 0x6, 0x53b301) src/pkg/testing/testing.go:510 +0x923 fp=0xc208019e78 sp=0xc208019cd8 testing.Main(0x5a89e8, 0x63cd80, 0x6, 0x6, 0x645ce0, 0x0, 0x0, 0x645ce0, 0x0, 0x0) src/pkg/testing/testing.go:440 +0x84 fp=0xc208019ef8 sp=0xc208019e78 main.main() runtime/pprof/_test/_testmain.go:57 +0x9c fp=0xc208019f50 sp=0xc208019ef8 runtime.main() src/pkg/runtime/proc.c:262 +0x121 fp=0xc208019fa8 sp=0xc208019f50 runtime.goexit() src/pkg/runtime/proc.c:1476 fp=0xc208019fb0 sp=0xc208019fa8 goroutine 2 [syscall]: runtime.notetsleepg(0xc208015f68, 0xffffffffffffffff) src/pkg/runtime/lock_futex.c:198 +0x4d fp=0xc208015f40 sp=0xc208015f10 runtime.MHeap_Scavenger() src/pkg/runtime/mheap.c:678 +0x13f fp=0xc208015fa8 sp=0xc208015f40 runtime.goexit() src/pkg/runtime/proc.c:1476 fp=0xc208015fb0 sp=0xc208015fa8 created by runtime.main src/pkg/runtime/proc.c:222 goroutine 3 [GC sweep wait]: runtime.park(0x417dc0, 0x641ed0, 0x6406da) src/pkg/runtime/proc.c:1407 +0x97 fp=0xc208031f70 sp=0xc208031f60 runtime.parkunlock(0x641ed0, 0x6406da) src/pkg/runtime/proc.c:1423 +0x3b fp=0xc208031f90 sp=0xc208031f70 bgsweep() src/pkg/runtime/mgc0.c:1047 +0x9a fp=0xc208031fa8 sp=0xc208031f90 runtime.goexit() src/pkg/runtime/proc.c:1476 fp=0xc208031fb0 sp=0xc208031fa8 created by runtime.gc src/pkg/runtime/mgc0.c:1255 goroutine 4 [finalizer wait, 8 minutes]: runtime.park(0x417dc0, 0x641ea8, 0x640882) src/pkg/runtime/proc.c:1407 +0x97 fp=0xc20802ff18 sp=0xc20802ff08 runtime.parkunlock(0x641ea8, 0x640882) src/pkg/runtime/proc.c:1423 +0x3b fp=0xc20802ff38 sp=0xc20802ff18 runfinq() src/pkg/runtime/mgc0.c:1613 +0xcf fp=0xc20802ffa8 sp=0xc20802ff38 runtime.goexit() src/pkg/runtime/proc.c:1476 fp=0xc20802ffb0 sp=0xc20802ffa8 created by runtime.gc src/pkg/runtime/mgc0.c:1255 goroutine 33 [running]: goroutine running on other thread; stack unavailable created by testing.RunTests src/pkg/testing/testing.go:509 +0x8db goroutine 10 [running]: goroutine running on other thread; stack unavailable created by runtime.MHeap_Scavenger src/pkg/runtime/mheap.c:641 rax 0xca rbx 0x642960 rcx 0xffffffffffffffff rdx 0x0 rdi 0x7f8fa6d19008 rsi 0x0 rbp 0x1 rsp 0xc2080afe38 r8 0x0 r9 0x0 r10 0x0 r11 0x286 r12 0xc20800c0c0 r13 0xc208028480 r14 0x0 r15 0x88 rip 0x42e271 rflags 0x286 cs 0x33 fs 0x0 gs 0x0 FAIL runtime/pprof 497.427sThe text was updated successfully, but these errors were encountered: