Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: SIGSEGV in runtime.shrinkstack #35916

Closed
junchih opened this issue Dec 1, 2019 · 5 comments
Closed

runtime: SIGSEGV in runtime.shrinkstack #35916

junchih opened this issue Dec 1, 2019 · 5 comments
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@junchih
Copy link

junchih commented Dec 1, 2019 

$ go version
go version go1.12.5 linux/amd64

I'm not sure about what I've done. Just get below panic sometime. And I've noticed that the 3rd number of runtime.memmove is a little weird, each time I've got this panic, this number is same.

Any idea why or any way how could I reproduce this kind of panic? Or any method how to debug that, I'll be appreciate it.

BTW, no Cgo function calling, no unsafe.Pointer misusing. I've checked that. And we are frequently using reflect package, including the function reflect.Select.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xbfffffffe0 pc=0x46cb04]
runtime stack:
runtime.throw(0x25c2e51, 0x2a)
    golang/go/src/runtime/panic.go:617 +0x72 fp=0x7f986c1ae880 sp=0x7f986c1ae850 pc=0x43ba92
runtime.sigpanic()
    golang/go/src/runtime/signal_unix.go:374 +0x4a9 fp=0x7f986c1ae8b0 sp=0x7f986c1ae880 pc=0x451669
runtime.memmove(0xc023ea9b08, 0xc00a66fb08, 0xffffffffffffc4f8)
    golang/go/src/runtime/memmove_amd64.s:500 +0x644 fp=0x7f986c1ae8b8 sp=0x7f986c1ae8b0 pc=0x46cb04
runtime.syncadjustsudogs(0xc00232de00, 0x4f8, 0x7f986c1ae980, 0xc00a66c000)
    golang/go/src/runtime/stack.go:797 +0x127 fp=0x7f986c1ae8f8 sp=0x7f986c1ae8b8 pc=0x454417
runtime.copystack(0xc00232de00, 0x2000, 0x412f800)
    golang/go/src/runtime/stack.go:858 +0x2d6 fp=0x7f986c1aeab0 sp=0x7f986c1ae8f8 pc=0x454716
runtime.shrinkstack(0xc00232de00)
    golang/go/src/runtime/stack.go:1137 +0x107 fp=0x7f986c1aead8 sp=0x7f986c1aeab0 pc=0x4555d7
runtime.scanstack(0xc00232de00, 0xc00005f770)
    golang/go/src/runtime/mgcmark.go:690 +0x90 fp=0x7f986c1aecd0 sp=0x7f986c1aead8 pc=0x42d180
runtime.scang(0xc00232de00, 0xc00005f770)
    golang/go/src/runtime/proc.go:888 +0x1e4 fp=0x7f986c1aed20 sp=0x7f986c1aecd0 pc=0x43f634
runtime.markroot.func1()
    golang/go/src/runtime/mgcmark.go:221 +0x6e fp=0x7f986c1aed60 sp=0x7f986c1aed20 pc=0x4669ae
runtime.markroot(0xc00005f770, 0x7f9800000191)
    golang/go/src/runtime/mgcmark.go:202 +0x2e9 fp=0x7f986c1aede0 sp=0x7f986c1aed60 pc=0x42c2c9
runtime.gcDrain(0xc00005f770, 0xb)
    golang/go/src/runtime/mgcmark.go:899 +0x112 fp=0x7f986c1aee38 sp=0x7f986c1aede0 pc=0x42daa2
runtime.gcBgMarkWorker.func2()
    golang/go/src/runtime/mgc.go:1924 +0x186 fp=0x7f986c1aee78 sp=0x7f986c1aee38 pc=0x466916
runtime.systemstack(0x0)
    golang/go/src/runtime/asm_amd64.s:351 +0x66 fp=0x7f986c1aee80 sp=0x7f986c1aee78 pc=0x469386
runtime.mstart()
    golang/go/src/runtime/proc.go:1153 fp=0x7f986c1aee88 sp=0x7f986c1aee80 pc=0x440040
@ALTree
Copy link
Member

ALTree commented Dec 1, 2019

Thanks for reporting this.

go version go1.12.5 linux/amd64

There has been eight 1.12 point-releases after 1.12.5, several of which fixed issues in the runtime, the compiler, and the linker.

Is your issue reproducible on the latest (go1.12.13)?

BTW, no Cgo function calling, no unsafe.Pointer misusing. I've checked that.

Does the program has any data race? If you run it with go run -race ..., does the race detector print any warnings?

@ALTree ALTree added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Dec 1, 2019
@junchih
Copy link
Author

junchih commented Dec 1, 2019

There has been eight 1.12 point-releases after 1.12.5, several of which fixed issues in the runtime, the compiler, and the linker.

Is your issue reproducible on the latest (go1.12.13)?

This issue can't be reproduced manually easily, this panic will occur about every 7 - 14 days uptime. I don't know how to reproduce it manually. So I'm not sure about if it could be reproduced at go 1.12.13. I'm still trying.

Does the program has any data race? If you run it with go run -race ..., does the race detector print any warnings?

I've detected the data racing, and still running the detecting. And for the reported data race issues, all I've fixed, and still panicing.

@junchih
Copy link
Author

junchih commented Dec 1, 2019

hi @ALTree , I even can not reproduce this issue with a simple main.go. So the code and the running environment should not be easily shared and modified for testing, sorry.

Any clue would be appreciated.

@randall77
Copy link
Contributor

Thanks for the trace.
The immediate problem is the 3rd arg to memmove as you noticed - it is negative.
That looks like it might be caused by a sudog which has an elem pointer that points below the current bottom of the stack (but still within the stack bounds). I don't see how that would happen.

We could paper over the problem by changing runtime.findsghi to trim to gp.sched.sp instead of stk.lo. That I think would guarantee a non-negative memmove size.
But that probably just papers over the real problem - why is there a sudog with a below-bottom-of-stack pointer? Without a reproducer it's going to be hard to track down.

@ALTree
Copy link
Member

ALTree commented Jun 18, 2021 •
edited
Loading

There hasn't been any news on this for almost 2 years, and we still don't have a reproducer, so I'm closing this issue. We can open a new issue if this turns out to be reproducible in a supported Go version.

@ALTree ALTree closed this as completed Jun 18, 2021
@golang golang locked and limited conversation to collaborators Jun 18, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ALTree @randall77 @gopherbot @junchih