runtime: systemstack called from unexpected goroutine/traceback has leftover defers #31538
Labels
FrozenDueToAge
NeedsInvestigation
Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
release-blocker
WaitingForInfo
Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Comments
|
Also I could provide the coredump file which's generated when |
|
Is there a way that we can reproduce the problem ourselves? |
ianlancetaylor
added
NeedsInvestigation
|
unfortunately still haven't worked out a minimal code which could reproduce the problem. But will keep trying to figure it out. |
andybons
added
the
WaitingForInfo
|
Hi @t57root, were you able to create a reproducer? |
|
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes, go version go1.12.4 linux/amd64
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
The program sometimes crashed with backtrace in stderr but sometimes crashed with nothing.
At first the code is golang mixed with some c code, which crashed with error
traceback has leftover defers, and the backtrace is attached as BT#1.Then I insert a dead-loop of sleep to the code before it running into the c code, and it crashed with error
systemstack called from unexpected goroutineSIGTRAP: trace trap. Backtrace is attached as BT#2.The crash only happens on two specific servers, and was never seen on other boxes before.
What did you expect to see?
Run without crash.
What did you see instead?
BT#1
runtime: g1: leftover defer sp=0x0 pc=0x0 defer 0xc0000b4000 sp=0x0 pc=0x0 fatal error: traceback has leftover defersruntime stack:
runtime.throw(0xaed3f5, 0x1d)
/usr/local/go/src/runtime/panic.go:617 +0x72
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000000180, 0x0, 0x0, 0x7fffffff, 0x7f1ac0ff8b80, 0x0, 0x0, ...)
/usr/local/go/src/runtime/traceback.go:563 +0x1c8c
runtime.scanstack(0xc000000180, 0xc000035270)
/usr/local/go/src/runtime/mgcmark.go:711 +0x15f
runtime.scang(0xc000000180, 0xc000035270)
/usr/local/go/src/runtime/proc.go:888 +0x1e4
runtime.markroot.func1()
/usr/local/go/src/runtime/mgcmark.go:221 +0x6e
runtime.markroot(0xc000035270, 0x5)
/usr/local/go/src/runtime/mgcmark.go:202 +0x2e9
runtime.gcDrainN(0xc000035270, 0x10000, 0x7f1accc34108)
/usr/local/go/src/runtime/mgcmark.go:1019 +0x100
runtime.gcAssistAlloc1(0xc000216900, 0x10000)
/usr/local/go/src/runtime/mgcmark.go:512 +0xf5
runtime.gcAssistAlloc.func1()
/usr/local/go/src/runtime/mgcmark.go:423 +0x33
runtime.systemstack(0x0)
/usr/local/go/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1153
goroutine 59 [GC assist marking]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc000276d40 sp=0xc000276d38 pc=0x45f500
runtime.gcAssistAlloc(0xc000216900)
/usr/local/go/src/runtime/mgcmark.go:422 +0x157 fp=0xc000276da0 sp=0xc000276d40 pc=0x426b57
runtime.mallocgc(0x6c, 0x0, 0x0, 0xc0003a2060)
/usr/local/go/src/runtime/malloc.go:843 +0x8e6 fp=0xc000276e40 sp=0xc000276da0 pc=0x414966
runtime.slicebytetostring(0x0, 0xc0004c0000, 0x6c, 0x600, 0x0, 0x0)
/usr/local/go/src/runtime/string.go:102 +0xa1 fp=0xc000276e70 sp=0xc000276e40 pc=0x44e461
zagen/monitor.getIo(0x2f0cb, 0xc0000b7f20)
/home/build/src/zagen/monitor/resource_reporter.go:128 +0xee fp=0xc000276f48 sp=0xc000276e70 pc=0x912ade
zagen/monitor.ResourceReport(0xadafa7, 0x6, 0xadb6af, 0x6, 0x2f0cb, 0x3c)
/home/build/src/zagen/monitor/resource_reporter.go:204 +0xba fp=0xc000276fb0 sp=0xc000276f48 pc=0x91359a
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000276fb8 sp=0xc000276fb0 pc=0x4615d1
created by main.MonitorEntry
/home/build/src/zagen/monitor.go:90 +0x7f6
goroutine 1 [syscall (scan)]:
zagen/monitor._Cfunc_limit_start(0x2f0cb, 0x0)
_cgo_gotypes.go:47 +0x4d
zagen/monitor.SigLimit(...)
/home/build/src/zagen/monitor/siglimit.go:350
main.MonitorEntry(0x7ffff16f1726, 0x17, 0xc0001c0000, 0x3, 0x0, 0xc00014e030, 0x29)
/home/build/src/zagen/monitor.go:94 +0x869
main.main()
/home/build/src/zagen/main.go:112 +0xa8c
goroutine 18 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x41
goroutine 10 [chan receive]:
zagen/rpc.(*Logger).Daemon(0xc00000ed00)
/home/build/src/zagen/rpc/log.go:138 +0x63
created by main.MonitorEntry
/home/build/src/zagen/monitor.go:38 +0x1ff
goroutine 57 [select]:
zagen/rpc.(*JsonRpc).Daemon(0xc00012a3c0)
/home/build/src/zagen/rpc/json.go:234 +0x135
created by main.MonitorEntry
/home/build/src/zagen/monitor.go:53 +0x407
goroutine 58 [select]:
zagen/rpc.(*Transporter).Work(0xc0002fa000)
/home/build/src/zagen/rpc/trans.go:84 +0xd5
created by main.MonitorEntry
/home/build/src/zagen/monitor.go:62 +0x540
goroutine 60 [runnable]:
runtime.SetFinalizer(0xa3afa0, 0xc0003df200, 0x0, 0x0)
/usr/local/go/src/runtime/mfinal.go:309 +0x812
os.(*file).close(0xc0003df200, 0xc0003df378, 0x0)
/usr/local/go/src/os/file_unix.go:256 +0x109
os.(*File).Close(0xc000232358, 0xc0003df320, 0xc00009fdd8)
/usr/local/go/src/os/file_unix.go:237 +0x33
os/exec.(*Cmd).closeDescriptors(0xc000304580, 0xc000246d10, 0x1, 0x1)
/usr/local/go/src/os/exec/exec.go:297 +0x5a
os/exec.(*Cmd).Wait(0xc000304580, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:487 +0x182
os/exec.(*Cmd).Run(0xc000304580, 0xc, 0xc00009fef8)
/usr/local/go/src/os/exec/exec.go:318 +0x5c
zagen/monitor.getCpu(0xadb6af, 0x6, 0x2f0cc, 0xc0003df140)
/home/build/src/zagen/monitor/resource_reporter.go:47 +0x137
zagen/monitor.ResourceReport(0xadcb17, 0x7, 0xadb6af, 0x6, 0x2f0cc, 0x3c)
/home/build/src/zagen/monitor/resource_reporter.go:202 +0x84
created by main.MonitorEntry
/home/build/src/zagen/monitor.go:91 +0x84f
BT#2
The text was updated successfully, but these errors were encountered: