Skip to content

runtime: "attempt to execute system stack code on user stack" during heap scavenging #34773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bcmills opened this issue Oct 8, 2019 · 7 comments
Closed

runtime: "attempt to execute system stack code on user stack" during heap scavenging #34773

bcmills opened this issue Oct 8, 2019 · 7 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.14

Comments

@bcmills
Copy link
Contributor

bcmills commented Oct 8, 2019

From the linux-amd64-nocgo builder (https://build.golang.org/log/abbb70348ec45659c8032defde341c61e90ec995):

fatal error: attempt to execute system stack code on user stack

goroutine 8 [running]:
runtime.throw(0x65ad83, 0x32)
	/workdir/go/src/runtime/panic.go:774 +0x72 fp=0xc000030610 sp=0xc0000305e0 pc=0x430562
runtime.morestackc()
	/workdir/go/src/runtime/stack.go:1315 +0x27 fp=0xc000030630 sp=0xc000030610 pc=0x44b117
runtime.stackfree(0xc000492000, 0xc000494000)
	/workdir/go/src/runtime/stack.go:422 +0x35c fp=0xc000030638 sp=0xc000030630 pc=0x448b2c
runtime.allocm(0xc00001e500, 0x65e5f8, 0x1)
	/workdir/go/src/runtime/proc.go:1517 +0xdb fp=0xc000030678 sp=0xc000030638 pc=0x43556b
runtime.newm(0x65e5f8, 0xc00001e500)
	/workdir/go/src/runtime/proc.go:1841 +0x39 fp=0xc0000306a8 sp=0xc000030678 pc=0x435d39
runtime.startm(0x0, 0xc000000c01)
	/workdir/go/src/runtime/proc.go:2006 +0x12a fp=0xc0000306d8 sp=0xc0000306a8 pc=0x43630a
runtime.wakep(...)
	/workdir/go/src/runtime/proc.go:2087
runtime.ready(0xc000000c00, 0x0, 0xd5731d01)
	/workdir/go/src/runtime/proc.go:666 +0x12f fp=0xc000030728 sp=0xc0000306d8 pc=0x43359f
runtime.wakeScavenger()
	/workdir/go/src/runtime/mgcscavenge.go:236 +0x85 fp=0xc000030750 sp=0xc000030728 pc=0x422d55
runtime.bgscavenge.func1(0x0, 0x0, 0x0)
	/workdir/go/src/runtime/mgcscavenge.go:295 +0x20 fp=0xc000030760 sp=0xc000030750 pc=0x45e6b0
runtime.timerproc(0x806140)
	/workdir/go/src/runtime/time.go:294 +0x6e fp=0xc0000307d8 sp=0xc000030760 pc=0x44fabe
runtime.goexit()
	/workdir/go/src/runtime/asm_amd64.s:1375 +0x1 fp=0xc0000307e0 sp=0xc0000307d8 pc=0x463cd1
created by runtime.(*timersBucket).addtimerLocked
	/workdir/go/src/runtime/time.go:166 +0x10e

The failure message is the same as in #30591, but the entry here seems to be through runtime.wakeScavenger rather than runtime.LockOSThread so the root cause may differ.

CC @mknyszek @aclements
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 8, 2019
@bcmills bcmills added this to the Go1.14 milestone Oct 8, 2019
@mknyszek
Copy link
Contributor

mknyszek commented Oct 8, 2019

Looking into it.

@mknyszek mknyszek self-assigned this Oct 8, 2019
@mknyszek
Copy link
Contributor

mknyszek commented Oct 8, 2019

OK so ready() needs to be called on the system stack.

Either wakeScavenger should be go:systemstack or ready should be goready. I'm not sure whether it actually matters which one we do. No other part of wakeScavenger needs the system stack and I have a slight preference for allowing it to be run mostly anywhere, though in some cases (like SetGCPercent) this means we'll go onto the system stack twice. It doesn't happen in any place that's particularly performance sensitive.

@aclements, @randall77, WDYT?

@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@randall77
Copy link
Contributor

I'd rather just put things on the system stack that need it. It serves as useful documentation, if nothing else. If that means we have to switch stacks twice for non-performance-sensitive code, so be it.

@randall77 randall77 modified the milestones: Backlog, Go1.14 Oct 14, 2019
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/200999 mentions this issue: runtime: call goready in wakeScavenger instead of ready

@lmm
Copy link

lmm commented Dec 12, 2019

@mknyszek @rsc do you know when this fix might appear in Go 1.13?

We're running into this occasionally in some integration tests. In what sorts of scenarios would this panic appear? (I don't know much about Golang GC). How does this panic get triggered?

@lmm lmm mentioned this issue Dec 13, 2019
Merged
nelljerram pushed a commit to nelljerram/felix that referenced this issue Dec 13, 2019
Use go-build v0.30, to revert to go 1.12
a14722c 
Because of golang/go#34773
@nelljerram nelljerram mentioned this issue Dec 13, 2019
Merged
@aclements
Copy link
Member

@gopherbot, please open a backport to 1.13.

@gopherbot gopherbot mentioned this issue Dec 13, 2019
Closed
@gopherbot
Copy link
Contributor

Backport issue(s) opened: #36127 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Dec 19, 2019
@golang golang locked and limited conversation to collaborators Dec 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.14
Development

No branches or pull requests
8 participants
@rsc @lmm @mknyszek @dmitshur @aclements @bcmills @randall77 @gopherbot