Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime/cgo: fatal error: runtime: split stack overflow #14067

Closed
mikioh opened this issue Jan 22, 2016 · 9 comments
Closed

runtime/cgo: fatal error: runtime: split stack overflow #14067

mikioh opened this issue Jan 22, 2016 · 9 comments
Labels
FrozenDueToAge OS-Dragonfly OS-NetBSD
Milestone
Go1.6Maybe

Comments

@mikioh
Copy link
Contributor

mikioh commented Jan 22, 2016

/cc @ianlancetaylor

git rev-parse: HEAD a4599ef on dragonfly 4.4-RELEASE x86_64.

TestCthread in misc/cgo/cgo_test.go crashes like the following:

cd $GOROOT/misc/cgo/test
go test -v run=TestCthread

runtime: newstack sp=0xc820034ea8 stack=[0xc820038000, 0xc820038fd0]
        morebuf={pc:0x4bcc48 sp:0xc820034eb8 lr:0x0}
        sched={pc:0x443dd0 sp:0xc820034eb0 lr:0x0 ctxt:0x0}
runtime: gp=0xc820000600, gp->status=0x4
 runtime: split stack overflow: 0xc820034ea8 < 0xc820038000
fatal error: runtime: split stack overflow

runtime stack:
runtime.throw(0x632cc0, 0x1d)
        /go/src/runtime/panic.go:530 +0x90
runtime.newstack()
        /go/src/runtime/stack.go:892 +0x899
runtime.morestack()
        /go/src/runtime/asm_amd64.s:359 +0x7f

goroutine 17 [stack growth, locked to thread]:
runtime.sigpanic()
        /go/src/runtime/sigpanic_unix.go:9 fp=0xc820034eb8 sp=0xc820034eb0
_/go/misc/cgo/test.Add(0x1)
        /go/misc/cgo/test/cthread.go:29 +0x68 fp=0xc820034ed0 sp=0xc820034eb8
_/go/misc/cgo/test._cgoexpwrap_bfd130227f24_Add(0x1)
        ??:0 +0x21 fp=0xc820034ee0 sp=0xc820034ed0
runtime.call32(0x0, 0x7ffffeff8f08, 0x7ffffeff8f90, 0x8)
        /go/src/runtime/asm_amd64.s:472 +0x3e fp=0xc820034f08 sp=0xc820034ee0
runtime.cgocallbackg1()
        /go/src/runtime/cgocall.go:267 +0x10c fp=0xc820034f40 sp=0xc820034f08
runtime.cgocallbackg()
        /go/src/runtime/cgocall.go:180 +0xd7 fp=0xc820034fa0 sp=0xc820034f40
runtime.cgocallback_gofunc(0x0, 0x0, 0x0)
        /go/src/runtime/asm_amd64.s:716 +0x60 fp=0xc820034fb0 sp=0xc820034fa0
runtime.goexit()
        /go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820034fb8 sp=0xc820034fb0

goroutine 1 [chan receive]:
testing.RunTests(0x652748, 0x713f00, 0x3d, 0x3d, 0x433601)
        /go/src/testing/testing.go:583 +0x8d2
testing.(*M).Run(0xc820049ef8, 0x406e95)
        /go/src/testing/testing.go:515 +0x81
main.main()
        _/go/misc/cgo/test/_test/_testmain.go:176 +0x117

goroutine 20 [syscall]:
os/signal.signal_recv(0x0)
        /go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
        /go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /go/src/os/signal/signal_unix.go:28 +0x37

goroutine 34 [syscall, locked to thread]:
_/go/misc/cgo/test._Cfunc_doAdd(0x60000000a)
        ??:0 +0x3a
_/go/misc/cgo/test.testCthread(0xc8200ce000)
        /go/misc/cgo/test/cthread.go:34 +0x39
_/go/misc/cgo/test.TestCthread(0xc8200ce000)
        /go/misc/cgo/test/cgo_test.go:39 +0x21
testing.tRunner(0xc8200ce000, 0x714170)
        /go/src/testing/testing.go:473 +0x98
created by testing.RunTests
        /go/src/testing/testing.go:582 +0x892

goroutine 50 [running, locked to thread]:
        goroutine running on other thread; stack unavailable

goroutine 51 [running, locked to thread]:
        goroutine running on other thread; stack unavailable

goroutine 52 [syscall, locked to thread]:
runtime.goexit()
        /go/src/runtime/asm_amd64.s:1998 +0x1
exit status 2

At the same time the kernel warns about the trapped frame corruption like the follwoing:

tail /var/log/message (or dmesg)

kernel: T_PAGEFLT: Warning %rip == 0
kernel: last message repeated 3 times
@bradfitz bradfitz added this to the Go1.6Maybe milestone Jan 22, 2016
@mikioh
Copy link
Contributor Author

mikioh commented Jan 22, 2016

It happens on NetBSD too.

@mikioh
Copy link
Contributor Author

mikioh commented Jan 22, 2016

It looks like this is the last missing piece for fixing signal stack setup issue introduced by a7cad52 in Go 1.6. The codepath goexit->crosscall2->cgocallback->cgocallback_gofunc->needm->minit on dragonfly and netbsd doesn't setup a new signal stack correctly.

@ianlancetaylor
Copy link
Contributor

I think this is another instance of the odd NetBSD/DragonFly behaviour: a thread created by pthread_create inherits the signal stack of the creating thread. This means that C code that calls pthread_create to create a thread that calls back into Go is broken. The Go code will use the existing signal stack, and there will be two thread sharing a signal stack, which can't work.

@ianlancetaylor
Copy link
Contributor

https://golang.org/cl/18835

@mikioh
Copy link
Contributor Author

mikioh commented Jan 22, 2016

FWIW, I'm just running https://go-review.googlesource.com/#/c/18834 on trybots. It just tests whether stack_t.ss_flags==0 in minit and I'm a bit surprised that the test fails even on Darwin or Linux. Perhaps we should not rely on the report from sigatlstack system call.

@ianlancetaylor
Copy link
Contributor

When a thread has a signal stack installed, and we call sigaltstack on that thread when we are not running in a signal handler, then it is normal for sigaltstack to return a stack_t with ss_flags == 0. It won't return SS_ONSTACK, because it is not actually running on the signal stack. And it won't return SS_DISABLE, because the signal stack is not disabled.

@mikioh
Copy link
Contributor Author

mikioh commented Jan 22, 2016

when we are not running in a signal handler...

I see. Thanks for the explanation.

@gopherbot
Copy link

CL https://golang.org/cl/18835 mentions this issue.

This was referenced Jan 22, 2016
Closed
Closed
@mikioh mikioh mentioned this issue Jul 12, 2016
Closed
@gopherbot
Copy link

CL https://golang.org/cl/29491 mentions this issue.

gopherbot pushed a commit that referenced this issue Sep 21, 2016
@cixtor
runtime: revert CL 18835; don't install new signal stack unconditiona…
5db80c3 
…lly on dragonfly

This change reverts CL 18835 which is a workaroud for older DragonFly
BSD kernels, and fixes #14051, #14052 and #14067 in a more general way
the same as other platforms except NetBSD.

This change also bumps the minimum required version of DragonFly BSD
kernel to 4.4.4.

Fixes #16329.

Change-Id: I0b44b6afa675f5ed9523914226bd9ec4809ba5ae
Reviewed-on: https://go-review.googlesource.com/29491
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@golang golang locked and limited conversation to collaborators Sep 21, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge OS-Dragonfly OS-NetBSD
Projects
None yet
Milestone
Go1.6Maybe
Development

No branches or pull requests
4 participants
@bradfitz @mikioh @ianlancetaylor @gopherbot