Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: STW GC stops working #34736

Closed
cherrymui opened this issue Oct 7, 2019 · 3 comments
Closed

runtime: STW GC stops working #34736

cherrymui opened this issue Oct 7, 2019 · 3 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@cherrymui
Copy link
Member

What version of Go are you using (go version)?

tip (cfe2320)

Does this issue reproduce with the latest release?

No. Go 1.13 works fine.

What operating system and processor architecture are you using (go env)?

darwin/amd64, linux/amd64

What did you do?

$ GODEBUG=gcstoptheworld=1 go tool compile hello.go

The compiler is just used as an example. This happens with essentially any non-trivial Go program. Setting GOGC=1 makes it happens more likely.

What did you expect to see?

Runs successfully.

What did you see instead?

$ GODEBUG=gcstoptheworld=1 go tool compile hello.go 
runtime: checkdead: find g 1 in status 1
fatal error: checkdead: runnable g

runtime stack:
runtime.throw(0x1a23d16, 0x15)
	/Users/cherryyz/src/go-tip/src/runtime/panic.go:774 +0x72
runtime.checkdead()
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:4273 +0x3a6
runtime.mput(...)
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:4664
runtime.stopm()
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:1963 +0x95
runtime.findrunnable(0xc000022500, 0x0)
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2428 +0x536
runtime.schedule()
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2561 +0x2be
runtime.park_m(0xc00047a000)
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2647 +0x9d
runtime.mcall(0x80000)
	/Users/cherryyz/src/go-tip/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [runnable]:
cmd/compile/internal/gc.initssaconfig()
	/Users/cherryyz/src/go-tip/src/cmd/compile/internal/gc/ssa.go:63 +0x308
cmd/compile/internal/gc.Main(0x1a3baa8)
	/Users/cherryyz/src/go-tip/src/cmd/compile/internal/gc/main.go:666 +0x2f94
main.main()
	/Users/cherryyz/src/go-tip/src/cmd/compile/main.go:51 +0xac

cc @aclements

@ALTree ALTree added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 7, 2019
@aclements
Copy link
Member

Some investigating;

cat >/tmp/hello.go <<EOF
package main
func main() { println("Hello") }
EOF
cat >/tmp/bisect.sh <<EOF
set -e
cd src
./make.bash
for i in 1 2 3; do
    GODEBUG=gcstoptheworld=1 GOGC=1 go tool compile -o /tmp/x /tmp/hello.go
done
EOF
chmod a+x /tmp/bisect.sh
git bisect start master $(git merge-base master go1.13)
git bisect run /tmp/bisect.sh

And the result:

7b294cdd8df0a9523010f6ffc80c59e64578f34b is the first bad commit
commit 7b294cdd8df0a9523010f6ffc80c59e64578f34b
Author: Michael Anthony Knyszek <mknyszek@google.com>
Date:   Mon Jun 17 19:03:09 2019 +0000

    runtime: don't hold worldsema across mark phase

cc @mknyszek

@mknyszek mknyszek added this to the Go1.14 milestone Oct 14, 2019
@mknyszek mknyszek self-assigned this Oct 14, 2019
@mknyszek
Copy link
Contributor

I believe I found the problem, and it's actually the deadlock detector detecting a deadlock! Unfortunately the deadlock happens during a STW GC so checkdead observes runnable goroutines and panics.

The problem is that after 7b294cdd8 we no longer hold onto worldsema for the duration of a GC; we hold gcsema instead. However, in STW GC we don't release it before calling Gosched on the user goroutine, but because of how STW GC is implemented we will need to STW again before the GC is complete, and we'll be unable to because this disabled user goroutine is holding onto worldsema. The fix is easy, just drop worldsema before calling Gosched.

@gopherbot
Copy link

Change https://golang.org/cl/208379 mentions this issue: runtime: release worldsema before Gosched in STW GC mode

@golang golang locked and limited conversation to collaborators Nov 21, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants