runtime: segfault during conservative scan of asynchronously preempted goroutine #39499
Comments
ALTree
changed the title
ALTree
added
the
NeedsInvestigation
|
This does look like corruption of internal runtime data structures. |
|
You mentioned possible misuse of unsafe. Could you compile with Can you reproduce this with |
|
Thank you for the information. I've got a build running with the requested |
|
We've been able to reproduce this issue on 1.14.4 with the For the time being I've attached the complete stack traces for reproductions that we have so far. |
|
I've got a reproduction of the segfault with the requested I'll leave the existing environment setup and update the issue whenever we come across any more reproductions. It's also worth noting that we have since downgraded a branch to Go 1.13.12 (and continued running testing) and we haven't yet encountered this issue. |
|
Just giving an update, we've just updated to Go 1.15 and will continue running our performance testing to see if we can reproduce the issue further (the performance cluster we were reproducing on recently had some issues/downtime). With that said, is there anything else we can provide to help debug the issue? |
gopherbot
added
the
compiler/runtime
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7f024c1505ab pc=0x4226d4]
runtime stack:
runtime.throw({0x11d71fc?, 0x1c97f9670?})
/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f01c97f9640 sp=0x7f01c97f9610 pc=0x439b5d
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x7f01c97f9690 sp=0x7f01c97f9640 pc=0x450409
runtime.(*mspan).isFree(...)
/usr/local/go/src/runtime/mbitmap.go:231
runtime.scanConservative(0xc00055c960, 0x1088, 0x0, 0x46dea1?, 0x7f01c97f9b68)
/usr/local/go/src/runtime/mgcmark.go:1432 +0x134 fp=0x7f01c97f96e8 sp=0x7f01c97f9690 pc=0x4226d4
runtime.scanframeworker(0x7f01c97f9a80, 0x7f01c97f9b68, 0x1?)
/usr/local/go/src/runtime/mgcmark.go:941 +0x148 fp=0x7f01c97f9750 sp=0x7f01c97f96e8 pc=0x421968
runtime.scanstack.func1(0x1ad08e0?, 0x1b47760?)
/usr/local/go/src/runtime/mgcmark.go:801 +0x25 fp=0x7f01c97f9778 sp=0x7f01c97f9750 pc=0x4217e5
runtime.gentraceback(0x20?, 0x1366128?, 0x0?, 0x7f01c97f9b68?, 0x0, 0x0, 0x7fffffff, 0x7f01c97f9cc8, 0x0?, 0x0)
/usr/local/go/src/runtime/traceback.go:334 +0xd0d fp=0x7f01c97f9ae8 sp=0x7f01c97f9778 pc=0x4602ed
runtime.scanstack(0xc0026489c0, 0xc000055740)
/usr/local/go/src/runtime/mgcmark.go:804 +0x1da fp=0x7f01c97f9cf0 sp=0x7f01c97f9ae8 pc=0x42125a
runtime.markroot.func1()
/usr/local/go/src/runtime/mgcmark.go:240 +0xc5 fp=0x7f01c97f9d40 sp=0x7f01c97f9cf0 pc=0x420085
runtime.markroot(0xc000055740, 0x11b6, 0x1)
/usr/local/go/src/runtime/mgcmark.go:213 +0x1a5 fp=0x7f01c97f9de0 sp=0x7f01c97f9d40 pc=0x41fd25
runtime.gcDrain(0xc000055740, 0x2)
/usr/local/go/src/runtime/mgcmark.go:1069 +0x39f fp=0x7f01c97f9e40 sp=0x7f01c97f9de0 pc=0x421dff
runtime.gcBgMarkWorker.func2()
/usr/local/go/src/runtime/mgc.go:1323 +0x154 fp=0x7f01c97f9e90 sp=0x7f01c97f9e40 pc=0x41e3d4
runtime.systemstack()
/usr/local/go/src/runtime/asm_amd64.s:492 +0x49 fp=0x7f01c97f9e98 sp=0x7f01c97f9e90 pc=0x46bce9@mknyszek we caught a panic whose stack trace similar to the above, the stack may be helpful. The go version is 1.19 |
|
@YuhangMa1117 Sorry, but Go 1.19 is no longer supported (https://go.dev/doc/devel/release#policy). Please try a newer version of Go -- if it still happens, feel free to reply back or file a new issue. Thanks. |
@mknyszek We caught a similar panic in 1.17.5/1.17.8. We are using cgo features, and we have shared memory calls in cgo such as mmap, etc., do these cause the final panic？ |
|
We ran into it here: vitessio/vitess#17456 in Go version 1.23.3. Is it possible for anybody here to write a brute force fuzzer that creates a lot of objects in many go-routines and let that run for hours/days? My Go knowledge is very low... |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes (but not consistently) - We have reproductions up to 1.14.3 (and have just
updated to 1.14.4 but no tests have been run as of yet).
What operating system and processor architecture are you using?
CentOS 7 amd64 - E5-2630 v2 (24 vCPU)
What issue are we seeing?
From a brief look at the stacktrace and runtime it looks like we are currently
seeing a segfault during the conservative scan of an asynchronously preempted
goroutine. While we have only seen this issue since updating to 1.14.1 (we
skipped 1.14) we do rely on a couple of libraries that make use of 'unsafe' so
we wouldn't be surprised if this was due to the misuse of 'unsafe' rather than
an issue with the runtime itself.
I've included the full stacktrace below along with a snippet from the same
backtrace displaying what I've described above. Any help debugging this issue
would be greatly appreciated whether that be tips on which GODEBUG settings to
use so that we can get more information about why this is happening or some
steps we could take to debug this issue/to provide you with extra information.
stack_trace.txt
The text was updated successfully, but these errors were encountered: