Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: timeout in TestNeedmDeadlock on darwin-amd64-10_12 builder #44303

Closed
bcmills opened this issue Feb 16, 2021 · 7 comments
Closed

runtime: timeout in TestNeedmDeadlock on darwin-amd64-10_12 builder #44303

bcmills opened this issue Feb 16, 2021 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@bcmills
Copy link
Contributor

bcmills commented Feb 16, 2021

2021-02-15T01:54:22-33d72fd/darwin-amd64-10_12
2020-10-29T15:13:09-50af50d/darwin-amd64-10_12

It looks like the testprogcgo process ends up wedged, but runBuildTestProg fails to terminate it so we don't get a useful stack trace.

See previously #42207 (the deadlock for which this test was added).
See also #43625 (TestNeedmDeadlock on MIPS).

goroutine 22881 [syscall, 2 minutes]:
syscall.syscall6(0x108d5e0, 0x19ef, 0xc0003bec74, 0x0, 0xc00007a090, 0x0, 0x0, 0x0, 0x0, 0x0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/sys_darwin.go:41 +0x2e fp=0xc0003bebc0 sp=0xc0003beb98 pc=0x107822e
syscall.wait4(0x19ef, 0xc0003bec74, 0x0, 0xc00007a090, 0x90, 0x1285a20, 0xc0003bed01)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/syscall/zsyscall_darwin_amd64.go:44 +0x85 fp=0xc0003bec38 sp=0xc0003bebc0 pc=0x108a345
syscall.Wait4(0x19ef, 0xc0003becc4, 0x0, 0xc00007a090, 0xc00010e420, 0x60, 0x100fd7b)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/syscall/syscall_bsd.go:144 +0x51 fp=0xc0003bec88 sp=0xc0003bec38 pc=0x1088af1
os.(*Process).wait(0xc000078030, 0xc00010e420, 0x12a5560, 0xc000080480)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/os/exec_unix.go:43 +0x85 fp=0xc0003becf8 sp=0xc0003bec88 pc=0x10a48c5
os.(*Process).Wait(...)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/os/exec.go:129
os/exec.(*Cmd).Wait(0xc0003c8000, 0x12a7ab8, 0xc00010e420)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/os/exec/exec.go:507 +0x65 fp=0xc0003bed78 sp=0xc0003becf8 pc=0x113b785
runtime_test.runBuiltTestProg(0xc000483500, 0xc00013e100, 0x71, 0x1291263, 0xd, 0x0, 0x0, 0x0, 0x0, 0x0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/crash_test.go:103 +0x2ed fp=0xc0003bee70 sp=0xc0003bed78 pc=0x11810cd
runtime_test.runTestProg(0xc000483500, 0x1290916, 0xb, 0x1291263, 0xd, 0x0, 0x0, 0x0, 0xc0321263d0, 0x321263d0010f64d0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/crash_test.go:57 +0x165 fp=0xc0003beef0 sp=0xc0003bee70 pc=0x1180d45
runtime_test.TestNeedmDeadlock(0xc000483500)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/crash_cgo_test.go:628 +0x7b fp=0xc0003bef80 sp=0xc0003beef0 pc=0x11809fb
testing.tRunner(0xc000483500, 0x12a7170)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/testing/testing.go:1194 +0xef fp=0xc0003befd0 sp=0xc0003bef80 pc=0x10f6b2f
runtime.goexit()
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0003befd8 sp=0xc0003befd0 pc=0x107b421
created by testing.(*T).Run
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/testing/testing.go:1239 +0x2b3
FAIL	runtime	180.025s

CC @ianlancetaylor @cherrymui
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 16, 2021
@bcmills bcmills added this to the Go1.17 milestone Feb 16, 2021
@bcmills
Copy link
Contributor Author

bcmills commented Feb 16, 2021 •
edited

Tentatively milestoning as a release-blocker for Go 1.17 until we better understand the root cause. This may be a deadlock in the runtime, and the failure mode would make it very difficult for a Go user to debug.

@dmitshur
Copy link
Contributor

CC @prattmic.

@dmitshur
Copy link
Contributor

dmitshur commented Apr 8, 2021

Friendly ping after 2 weeks since it's a release blocking issue for Go 1.17.

@ianlancetaylor
Copy link
Contributor

There seem to only be two failures since the test was added in October, 2020. Both failures are on Darwin 10.12. In both failures the test hits a 3 minute timeout, which is interesting because

  1. The test program (in runtime/testdata/testprogcgo/needmdeadlock.go) itself has a 5 second timeout before exiting, and
  2. the test runner (runBuiltTestProg in runtime/crash_test.go) has a 1 minute timeout before sending a SIGQUIT signal.

It seems that the test program has deadlocked so hard that both of those timeouts fail to stop it.

@ianlancetaylor
Copy link
Contributor

This has only happened twice, it's only happened on macOS 10.12, and we are dropping support for macOS 10.12 in 1.17 anyhow. Removing the release-blocker label.

@ianlancetaylor ianlancetaylor modified the milestones: Go1.17, Backlog Apr 15, 2021
@gopherbot
Copy link

Change https://golang.org/cl/312692 mentions this issue: dashboard: limit macOS 10.12 builders to at most go1.16

gopherbot pushed a commit to golang/build that referenced this issue Apr 22, 2021
@cagedmantis
dashboard: limit macOS 10.12 builders to at most go1.16
6469a76 
macOS will not be supported after go1.16. This change limits the macOS
10.12 builders to build previous to go1.17.

Updates golang/go#44303
For golang/go#23011

Change-Id: Ib6b4e7939ffb9032b255574d3f0b4fffaad97803
Reviewed-on: https://go-review.googlesource.com/c/build/+/312692
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@dmitshur
Copy link
Contributor

macOS 10.12 is no longer supported by any supported Go release, and the builder was dropped in CL 419194. Let's close this @bcmills?

@bcmills bcmills closed this as not planned Won't fix, can't repro, duplicate, stale Sep 23, 2022
@golang golang locked and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@dmitshur @ianlancetaylor @bcmills @gopherbot