Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: go test deadlocked without enforcing timeouts when killed with ^C #60203

Closed
bcmills opened this issue May 15, 2023 · 7 comments
Closed

cmd/go: go test deadlocked without enforcing timeouts when killed with ^C #60203

bcmills opened this issue May 15, 2023 · 7 comments
Assignees
@bcmills
Labels
GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.21

Comments

@bcmills
Copy link
Contributor

bcmills commented May 15, 2023

What version of Go are you using (go version)?

~/go/src$ go version
go version devel go1.21-902e37757a Wed May 10 08:13:36 2023 -0400 linux/amd64

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/usr/local/google/home/bcmills/.cache/go-build'
GOENV='/usr/local/google/home/bcmills/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/usr/local/google/home/bcmills/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/usr/local/google/home/bcmills'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/google/home/bcmills/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLDIR='/usr/local/google/home/bcmills/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.21-902e37757a Wed May 10 08:13:36 2023 -0400'
GCCGO='/usr/bin/gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/usr/local/google/home/bcmills/go/src/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1247785885=/tmp/go-build -gno-record-gcc-switches'

What did you do?

go test cmd/go/..., then try to kill the test by sending ^C from the terminal session.

What did you expect to see?

Ideally: test terminates ~immediately.
Acceptable: test terminates after a short delay.
Marginally ok: test completes, fails, or times out as usual.

What did you see instead?

The go test process deadlocked for ~20h, when the test timeout is normally 10m.

go test cmd/go/...
SIGABRT: abort
PC=0x46c681 m=13 sigcode=0

goroutine 0 [idle]:
runtime.futex()
        /usr/local/google/home/bcmills/go/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0xc00068dd98 sp=0xc00068dd90 pc=0x46c681
runtime.futexsleep(0x46c26d?, 0x68de18?, 0x444727?)
        /usr/local/google/home/bcmills/go/src/runtime/os_linux.go:69 +0x30 fp=0xc00068dde8 sp=0xc00068dd98 pc=0x432150
runtime.notesleep(0xc000680548)
        /usr/local/google/home/bcmills/go/src/runtime/lock_futex.go:160 +0x87 fp=0xc00068de20 sp=0xc00068dde8 pc=0x40b327
runtime.mPark(...)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:1538
runtime.stopm()
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:2374 +0x8c fp=0xc00068de50 sp=0xc00068de20 pc=0x43c5ac
runtime.findRunnable()
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:3064 +0xa7c fp=0xc00068df58 sp=0xc00068de50 pc=0x43dddc
runtime.schedule()
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:3417 +0xb1 fp=0xc00068df90 sp=0xc00068df58 pc=0x43eb71
runtime.park_m(0xc00074ad00?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:3568 +0x119 fp=0xc00068dfc0 sp=0xc00068df90 pc=0x43f039
runtime.mcall()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:458 +0x4e fp=0xc00068dfd8 sp=0xc00068dfc0 pc=0x4689ce

goroutine 1 [semacquire, 1209 minutes]:
runtime.gopark(0x10?, 0x10?, 0x40?, 0x2?, 0x8c39f9?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000c45508 sp=0xc000c454e8 pc=0x43890e
runtime.goparkunlock(...)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:393
runtime.semacquire1(0xc0006a0248, 0x30?, 0x1, 0x0, 0x6b?)
        /usr/local/google/home/bcmills/go/src/runtime/sema.go:160 +0x20f fp=0xc000c45578 sp=0xc000c45508 pc=0x44980f
sync.runtime_Semacquire(0xc000a94c00?)
        /usr/local/google/home/bcmills/go/src/runtime/sema.go:62 +0x25 fp=0xc000c455b0 sp=0xc000c45578 pc=0x466d25
sync.(*WaitGroup).Wait(0x0?)
        /usr/local/google/home/bcmills/go/src/sync/waitgroup.go:116 +0x48 fp=0xc000c455d8 sp=0xc000c455b0 pc=0x476168
cmd/go/internal/work.(*Builder).Do(0xc000722370, {0xb4dd18, 0xece0c0}, 0xc000c3de40)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:227 +0x49e fp=0xc000c456f0 sp=0xc000c455d8 pc=0x8d0fde
cmd/go/internal/test.runTest({0xb4dd18, 0xece0c0}, 0xc0000284e0?, {0xc0000221d0?, 0x7f3ba6e18108?, 0x3?})
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:954 +0x1a17 fp=0xc000c45d20 sp=0xc000c456f0 pc=0x934317
main.invoke(0xe93340, {0xc0000221c0, 0x2, 0x2})
        /usr/local/google/home/bcmills/go/src/cmd/go/main.go:243 +0x4d6 fp=0xc000c45e10 sp=0xc000c45d20 pc=0x94d016
main.main()
        /usr/local/google/home/bcmills/go/src/cmd/go/main.go:183 +0x7d1 fp=0xc000c45f40 sp=0xc000c45e10 pc=0x94c811
runtime.main()
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:256 +0x2b2 fp=0xc000c45fe0 sp=0xc000c45f40 pc=0x4384d2
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000c45fe8 sp=0xc000c45fe0 pc=0x46a841

goroutine 2 [force gc (idle), 2 minutes]:
runtime.gopark(0x107268b4fdd24?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000074fa8 sp=0xc000074f88 pc=0x43890e
runtime.goparkunlock(...)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:393
runtime.forcegchelper()
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:311 +0xb3 fp=0xc000074fe0 sp=0xc000074fa8 pc=0x438793
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000074fe8 sp=0xc000074fe0 pc=0x46a841
created by runtime.init.6 in goroutine 1
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:299 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0xe9d901?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000075778 sp=0xc000075758 pc=0x43890e
runtime.goparkunlock(...)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:393
runtime.bgsweep(0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/mgcsweep.go:321 +0xdf fp=0xc0000757c8 sp=0xc000075778 pc=0x424b9f
runtime.gcenable.func1()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:178 +0x25 fp=0xc0000757e0 sp=0xc0000757c8 pc=0x419c05
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0000757e8 sp=0xc0000757e0 pc=0x46a841
created by runtime.gcenable in goroutine 1
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:178 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xe9e240?, 0xe9e240?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000075f70 sp=0xc000075f50 pc=0x43890e
runtime.goparkunlock(...)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:393
runtime.(*scavengerState).park(0xe9e240)
        /usr/local/google/home/bcmills/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000075fa0 sp=0xc000075f70 pc=0x422389
runtime.bgscavenge(0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000075fc8 sp=0xc000075fa0 pc=0x422939
runtime.gcenable.func2()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:179 +0x25 fp=0xc000075fe0 sp=0xc000075fc8 pc=0x419ba5
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000075fe8 sp=0xc000075fe0 pc=0x46a841
created by runtime.gcenable in goroutine 1
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:179 +0xa5

goroutine 5 [finalizer wait, 1207 minutes]:
runtime.gopark(0x0?, 0xab0b50?, 0x40?, 0x1?, 0x2000000020?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000074628 sp=0xc000074608 pc=0x43890e
runtime.runfinq()
        /usr/local/google/home/bcmills/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000747e0 sp=0xc000074628 pc=0x418c87
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0000747e8 sp=0xc0000747e0 pc=0x46a841
created by runtime.createfing in goroutine 1
        /usr/local/google/home/bcmills/go/src/runtime/mfinal.go:163 +0x3d

goroutine 2820 [chan receive, 1209 minutes]:
runtime.gopark(0x8d428c?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0003938b0 sp=0xc000393890 pc=0x43890e
runtime.chanrecv(0xc0002629c0, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000393928 sp=0xc0003938b0 pc=0x4060ad
runtime.chanrecv1(0x4f4a58?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000393950 sp=0xc000393928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc00081e8a0, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000ba0b00)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000393d70 sp=0xc000393950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000ba0b00)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000393f20 sp=0xc000393d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000393fe0 sp=0xc000393f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000393fe8 sp=0xc000393fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2818 [chan receive, 1209 minutes]:
runtime.gopark(0x7f3b5ff31628?, 0x0?, 0x28?, 0x19?, 0x40cb85?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000ee18b0 sp=0xc000ee1890 pc=0x43890e
runtime.chanrecv(0xc0002630e0, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000ee1928 sp=0xc000ee18b0 pc=0x4060ad
runtime.chanrecv1(0xc000ee19c0?, 0xc000ee1990?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000ee1950 sp=0xc000ee1928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc0006a7980, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b30420)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000ee1d70 sp=0xc000ee1950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b30420)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000ee1f20 sp=0xc000ee1d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000ee1fe0 sp=0xc000ee1f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000ee1fe8 sp=0xc000ee1fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2822 [chan receive, 1209 minutes]:
runtime.gopark(0xc00021f8c8?, 0xc00021f928?, 0x28?, 0xf9?, 0x40cb85?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00021f8b0 sp=0xc00021f890 pc=0x43890e
runtime.chanrecv(0xc0002635c0, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc00021f928 sp=0xc00021f8b0 pc=0x4060ad
runtime.chanrecv1(0xc00021f9c0?, 0xc00021f990?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc00021f950 sp=0xc00021f928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc0002833e0, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b658c0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc00021fd70 sp=0xc00021f950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b658c0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc00021ff20 sp=0xc00021fd70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc00021ffe0 sp=0xc00021ff20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00021ffe8 sp=0xc00021ffe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2828 [chan receive, 1209 minutes]:
runtime.gopark(0xc00128f8c8?, 0xc00128f928?, 0x28?, 0xf9?, 0x40cb85?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00128f8b0 sp=0xc00128f890 pc=0x43890e
runtime.chanrecv(0xc000262f00, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc00128f928 sp=0xc00128f8b0 pc=0x4060ad
runtime.chanrecv1(0xc00128f9c0?, 0xc00128f990?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc00128f950 sp=0xc00128f928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc000223ec0, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b1c840)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc00128fd70 sp=0xc00128f950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b1c840)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc00128ff20 sp=0xc00128fd70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc00128ffe0 sp=0xc00128ff20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00128ffe8 sp=0xc00128ffe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 65 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc0001f57d0?, 0x5c96bc?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0001f5750 sp=0xc0001f5730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0001f57e0 sp=0xc0001f5750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0001f57e8 sp=0xc0001f57e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 81 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc0001fafd0?, 0x5c96bc?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0001faf50 sp=0xc0001faf30 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0001fafe0 sp=0xc0001faf50 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0001fafe8 sp=0xc0001fafe0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 82 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc0001fa7d0?, 0x5c96bc?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0001fa750 sp=0xc0001fa730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0001fa7e0 sp=0xc0001fa750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0001fa7e8 sp=0xc0001fa7e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 97 [GC worker (idle), 1209 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00079a750 sp=0xc00079a730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc00079a7e0 sp=0xc00079a750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00079a7e8 sp=0xc00079a7e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 113 [GC worker (idle), 1209 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000796750 sp=0xc000796730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0007967e0 sp=0xc000796750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0007967e8 sp=0xc0007967e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 83 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc0001f8fd0?, 0x5c96bc?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0001f8f50 sp=0xc0001f8f30 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0001f8fe0 sp=0xc0001f8f50 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0001f8fe8 sp=0xc0001f8fe0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 84 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc0001df7d0?, 0x5c96bc?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0001df750 sp=0xc0001df730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0001df7e0 sp=0xc0001df750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0001df7e8 sp=0xc0001df7e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 85 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc0001dffd0?, 0x5c96bc?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0001dff50 sp=0xc0001dff30 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0001dffe0 sp=0xc0001dff50 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0001dffe8 sp=0xc0001dffe0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 114 [GC worker (idle), 2 minutes]:
runtime.gopark(0xecf400?, 0x1?, 0xd6?, 0x9c?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000796f50 sp=0xc000796f30 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc000796fe0 sp=0xc000796f50 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000796fe8 sp=0xc000796fe0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 86 [GC worker (idle), 2 minutes]:
runtime.gopark(0x107268b71cec0?, 0x1?, 0xd5?, 0xb7?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0001dd750 sp=0xc0001dd730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0001dd7e0 sp=0xc0001dd750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0001dd7e8 sp=0xc0001dd7e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 87 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc533ee9ed7a2?, 0x1?, 0xda?, 0x1a?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0001e0f50 sp=0xc0001e0f30 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0001e0fe0 sp=0xc0001e0f50 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0001e0fe8 sp=0xc0001e0fe0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 98 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc533eea0cca6?, 0x3?, 0xa4?, 0x1b?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00079af50 sp=0xc00079af30 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc00079afe0 sp=0xc00079af50 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00079afe8 sp=0xc00079afe0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 99 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc532e89c6643?, 0x3?, 0x8b?, 0x77?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00079b750 sp=0xc00079b730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc00079b7e0 sp=0xc00079b750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00079b7e8 sp=0xc00079b7e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 115 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc5341e42b08d?, 0x1?, 0x72?, 0x2f?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000797750 sp=0xc000797730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0007977e0 sp=0xc000797750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0007977e8 sp=0xc0007977e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 100 [GC worker (idle), 2 minutes]:
runtime.gopark(0xecf400?, 0x1?, 0x3a?, 0x87?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00079bf50 sp=0xc00079bf30 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc00079bfe0 sp=0xc00079bf50 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00079bfe8 sp=0xc00079bfe0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 88 [GC worker (idle)]:
runtime.gopark(0xecf400?, 0x1?, 0xa2?, 0x57?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0001e1750 sp=0xc0001e1730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0001e17e0 sp=0xc0001e1750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0001e17e8 sp=0xc0001e17e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 116 [GC worker (idle)]:
runtime.gopark(0xecf400?, 0x1?, 0xa8?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000797f50 sp=0xc000797f30 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc000797fe0 sp=0xc000797f50 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000797fe8 sp=0xc000797fe0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 101 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc532e89ca27f?, 0x1?, 0x58?, 0x35?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00079c750 sp=0xc00079c730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc00079c7e0 sp=0xc00079c750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00079c7e8 sp=0xc00079c7e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 117 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xecf400?, 0x3?, 0xa8?, 0x17?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000798750 sp=0xc000798730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0007987e0 sp=0xc000798750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0007987e8 sp=0xc0007987e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 102 [GC worker (idle), 1109 minutes]:
runtime.gopark(0xecf400?, 0x1?, 0x16?, 0x3e?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00079cf50 sp=0xc00079cf30 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc00079cfe0 sp=0xc00079cf50 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00079cfe8 sp=0xc00079cfe0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 89 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xecf400?, 0x1?, 0x73?, 0x5c?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000888750 sp=0xc000888730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0008887e0 sp=0xc000888750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0008887e8 sp=0xc0008887e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 90 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xc532e89ca211?, 0x3?, 0xa8?, 0xf5?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000888f50 sp=0xc000888f30 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc000888fe0 sp=0xc000888f50 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000888fe8 sp=0xc000888fe0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 103 [GC worker (idle)]:
runtime.gopark(0x107268b7361ea?, 0x1?, 0xcd?, 0xa7?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00079d750 sp=0xc00079d730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc00079d7e0 sp=0xc00079d750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00079d7e8 sp=0xc00079d7e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 91 [GC worker (idle), 1209 minutes]:
runtime.gopark(0xecf400?, 0x1?, 0x92?, 0x83?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000889750 sp=0xc000889730 pc=0x43890e
runtime.gcBgMarkWorker()
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1297 +0xe5 fp=0xc0008897e0 sp=0xc000889750 pc=0x41b865
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0008897e8 sp=0xc0008897e0 pc=0x46a841
created by runtime.gcBgMarkStartWorkers in goroutine 44
        /usr/local/google/home/bcmills/go/src/runtime/mgc.go:1221 +0x1c

goroutine 2826 [chan receive, 1209 minutes]:
runtime.gopark(0xc00128d8c8?, 0xc00128d928?, 0x28?, 0xd9?, 0x40cb85?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00128d8b0 sp=0xc00128d890 pc=0x43890e
runtime.chanrecv(0xc000263260, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc00128d928 sp=0xc00128d8b0 pc=0x4060ad
runtime.chanrecv1(0xc00128d9c0?, 0xc00128d990?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc00128d950 sp=0xc00128d928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc0006d3740, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b42f20)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc00128dd70 sp=0xc00128d950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b42f20)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc00128df20 sp=0xc00128dd70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc00128dfe0 sp=0xc00128df20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00128dfe8 sp=0xc00128dfe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2824 [chan receive, 1209 minutes]:
runtime.gopark(0xb45ff8?, 0x1?, 0x40?, 0xe1?, 0xc0009c4200?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000edd8b0 sp=0xc000edd890 pc=0x43890e
runtime.chanrecv(0xc000263800, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000edd928 sp=0xc000edd8b0 pc=0x4060ad
runtime.chanrecv1(0xc000edd998?, 0x4403d1?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000edd950 sp=0xc000edd928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc000283800, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b886e0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000eddd70 sp=0xc000edd950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b886e0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000eddf20 sp=0xc000eddd70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000eddfe0 sp=0xc000eddf20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000eddfe8 sp=0xc000eddfe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2835 [chan receive, 1209 minutes]:
runtime.gopark(0x776f6c666c727463?, 0x72757465526f6e2e?, 0x6e?, 0xff?, 0x692f6f7470797263?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00060e8b0 sp=0xc00060e890 pc=0x43890e
runtime.chanrecv(0xc000262900, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc00060e928 sp=0xc00060e8b0 pc=0x4060ad
runtime.chanrecv1(0x616e7265746e692f?, 0x676e69726f622f6c?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc00060e950 sp=0xc00060e928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc00081e7e0, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000ba0580)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc00060ed70 sp=0xc00060e950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000ba0580)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc00060ef20 sp=0xc00060ed70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc00060efe0 sp=0xc00060ef20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00060efe8 sp=0xc00060efe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2821 [chan receive, 1209 minutes]:
runtime.gopark(0xb45ff8?, 0x1?, 0x40?, 0xe1?, 0xc00107b700?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000edf8b0 sp=0xc000edf890 pc=0x43890e
runtime.chanrecv(0xc0002639e0, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000edf928 sp=0xc000edf8b0 pc=0x4060ad
runtime.chanrecv1(0xc000edf998?, 0x4403d1?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000edf950 sp=0xc000edf928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc0005a4540, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b10420)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000edfd70 sp=0xc000edf950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b10420)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000edff20 sp=0xc000edfd70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000edffe0 sp=0xc000edff20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000edffe8 sp=0xc000edffe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2832 [chan receive, 1209 minutes]:
runtime.gopark(0xb45ff8?, 0x1?, 0x40?, 0xe1?, 0xc000ab8300?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000c638b0 sp=0xc000c63890 pc=0x43890e
runtime.chanrecv(0xc0002631a0, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000c63928 sp=0xc000c638b0 pc=0x4060ad
runtime.chanrecv1(0xc000c63998?, 0x4403d1?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000c63950 sp=0xc000c63928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc0002825a0, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b378c0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000c63d70 sp=0xc000c63950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b378c0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000c63f20 sp=0xc000c63d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000c63fe0 sp=0xc000c63f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000c63fe8 sp=0xc000c63fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2829 [chan receive, 1209 minutes]:
runtime.gopark(0xc000c4b8c8?, 0xc000c4b928?, 0x28?, 0xb9?, 0x40cb85?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000c4b8b0 sp=0xc000c4b890 pc=0x43890e
runtime.chanrecv(0xc000262a20, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000c4b928 sp=0xc000c4b8b0 pc=0x4060ad
runtime.chanrecv1(0xc000c4b9c0?, 0xc000c4b990?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000c4b950 sp=0xc000c4b928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc00081e900, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000ba0dc0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000c4bd70 sp=0xc000c4b950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000ba0dc0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000c4bf20 sp=0xc000c4bd70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000c4bfe0 sp=0xc000c4bf20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000c4bfe8 sp=0xc000c4bfe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2830 [chan receive, 1209 minutes]:
runtime.gopark(0xb45ff8?, 0x1?, 0x40?, 0xe1?, 0xc0000c2d00?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00021d8b0 sp=0xc00021d890 pc=0x43890e
runtime.chanrecv(0xc0002627e0, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc00021d928 sp=0xc00021d8b0 pc=0x4060ad
runtime.chanrecv1(0xc00021d998?, 0x4403d1?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc00021d950 sp=0xc00021d928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc00081e120, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000c43760)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc00021dd70 sp=0xc00021d950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000c43760)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc00021df20 sp=0xc00021dd70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc00021dfe0 sp=0xc00021df20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00021dfe8 sp=0xc00021dfe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2837 [chan receive, 1209 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc00111e8b0 sp=0xc00111e890 pc=0x43890e
runtime.chanrecv(0xc000263140, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc00111e928 sp=0xc00111e8b0 pc=0x4060ad
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc00111e950 sp=0xc00111e928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc000282420, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b36840)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc00111ed70 sp=0xc00111e950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b36840)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc00111ef20 sp=0xc00111ed70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc00111efe0 sp=0xc00111ef20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00111efe8 sp=0xc00111efe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2827 [chan receive, 1209 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0006108b0 sp=0xc000610890 pc=0x43890e
runtime.chanrecv(0xc000263560, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000610928 sp=0xc0006108b0 pc=0x4060ad
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000610950 sp=0xc000610928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc000282d80, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b3dce0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000610d70 sp=0xc000610950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b3dce0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000610f20 sp=0xc000610d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000610fe0 sp=0xc000610f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000610fe8 sp=0xc000610fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2825 [chan receive, 1209 minutes]:
runtime.gopark(0xb45ff8?, 0x1?, 0x40?, 0xe1?, 0xc000b08680?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0002af8b0 sp=0xc0002af890 pc=0x43890e
runtime.chanrecv(0xc000263620, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc0002af928 sp=0xc0002af8b0 pc=0x4060ad
runtime.chanrecv1(0xc0002af998?, 0x4403d1?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc0002af950 sp=0xc0002af928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc0002834a0, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000c20840)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc0002afd70 sp=0xc0002af950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000c20840)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc0002aff20 sp=0xc0002afd70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc0002affe0 sp=0xc0002aff20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0002affe8 sp=0xc0002affe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2834 [chan receive, 1209 minutes]:
runtime.gopark(0xc0002b18c8?, 0xc0002b1928?, 0x28?, 0x19?, 0x40cb85?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0002b18b0 sp=0xc0002b1890 pc=0x43890e
runtime.chanrecv(0xc0002628a0, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc0002b1928 sp=0xc0002b18b0 pc=0x4060ad
runtime.chanrecv1(0xc0002b19c0?, 0xc0002b1990?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc0002b1950 sp=0xc0002b1928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc00081e780, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000879ce0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc0002b1d70 sp=0xc0002b1950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000879ce0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc0002b1f20 sp=0xc0002b1d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc0002b1fe0 sp=0xc0002b1f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0002b1fe8 sp=0xc0002b1fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2817 [chan receive, 1209 minutes]:
runtime.gopark(0xb45ff8?, 0x1?, 0x40?, 0xe1?, 0xc000b82700?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000ee38b0 sp=0xc000ee3890 pc=0x43890e
runtime.chanrecv(0xc000263440, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000ee3928 sp=0xc000ee38b0 pc=0x4060ad
runtime.chanrecv1(0xc000ee3998?, 0x4403d1?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000ee3950 sp=0xc000ee3928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc000327b60, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b7d4a0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000ee3d70 sp=0xc000ee3950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b7d4a0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000ee3f20 sp=0xc000ee3d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000ee3fe0 sp=0xc000ee3f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000ee3fe8 sp=0xc000ee3fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2833 [chan receive, 1209 minutes]:
runtime.gopark(0xb45ff8?, 0x1?, 0x40?, 0xe1?, 0xc000835280?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0012918b0 sp=0xc001291890 pc=0x43890e
runtime.chanrecv(0xc0002636e0, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc001291928 sp=0xc0012918b0 pc=0x4060ad
runtime.chanrecv1(0xc001291998?, 0x4403d1?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc001291950 sp=0xc001291928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc0002835c0, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000c211e0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc001291d70 sp=0xc001291950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000c211e0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc001291f20 sp=0xc001291d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc001291fe0 sp=0xc001291f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc001291fe8 sp=0xc001291fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2831 [chan receive, 1209 minutes]:
runtime.gopark(0xb45ff8?, 0x1?, 0x40?, 0xe1?, 0xc000a99800?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000cc18b0 sp=0xc000cc1890 pc=0x43890e
runtime.chanrecv(0xc000263920, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000cc1928 sp=0xc000cc18b0 pc=0x4060ad
runtime.chanrecv1(0xc000cc1998?, 0x4403d1?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000cc1950 sp=0xc000cc1928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc0005a4480, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc0006ee6e0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000cc1d70 sp=0xc000cc1950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc0006ee6e0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000cc1f20 sp=0xc000cc1d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000cc1fe0 sp=0xc000cc1f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000cc1fe8 sp=0xc000cc1fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 3634 [syscall]:
runtime.notetsleepg(0xffffffffffffffff?, 0xc000d59f28?)
        /usr/local/google/home/bcmills/go/src/runtime/lock_futex.go:236 +0x29 fp=0xc000d59fa0 sp=0xc000d59f68 pc=0x40b609
os/signal.signal_recv()
        /usr/local/google/home/bcmills/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000d59fc0 sp=0xc000d59fa0 pc=0x467489
os/signal.loop()
        /usr/local/google/home/bcmills/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000d59fe0 sp=0xc000d59fc0 pc=0x5ae0d3
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000d59fe8 sp=0xc000d59fe0 pc=0x46a841
created by os/signal.Notify.func1.1 in goroutine 2816
        /usr/local/google/home/bcmills/go/src/os/signal/signal.go:151 +0x1f

goroutine 3594 [select, 1209 minutes, locked to thread]:
runtime.gopark(0xc000070fa8?, 0x2?, 0x0?, 0x0?, 0xc000070fa4?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000070e40 sp=0xc000070e20 pc=0x43890e
runtime.selectgo(0xc000070fa8, 0xc000070fa0, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/select.go:327 +0x725 fp=0xc000070f58 sp=0xc000070e40 pc=0x4487e5
runtime.ensureSigM.func1()
        /usr/local/google/home/bcmills/go/src/runtime/signal_unix.go:994 +0x1a5 fp=0xc000070fe0 sp=0xc000070f58 pc=0x461e05
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000070fe8 sp=0xc000070fe0 pc=0x46a841
created by runtime.ensureSigM in goroutine 2816
        /usr/local/google/home/bcmills/go/src/runtime/signal_unix.go:977 +0xc8

goroutine 2819 [chan receive, 1209 minutes]:
runtime.gopark(0xb45ff8?, 0x1?, 0x40?, 0xe1?, 0xc000a99400?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0007938b0 sp=0xc000793890 pc=0x43890e
runtime.chanrecv(0xc000263b60, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000793928 sp=0xc0007938b0 pc=0x4060ad
runtime.chanrecv1(0xc000793998?, 0x4403d1?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000793950 sp=0xc000793928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc00042d980, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b702c0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000793d70 sp=0xc000793950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b702c0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000793f20 sp=0xc000793d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000793fe0 sp=0xc000793f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000793fe8 sp=0xc000793fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2836 [chan receive, 1209 minutes]:
runtime.gopark(0x8d428c?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0002218b0 sp=0xc000221890 pc=0x43890e
runtime.chanrecv(0xc000263980, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000221928 sp=0xc0002218b0 pc=0x4060ad
runtime.chanrecv1(0xc000221998?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000221950 sp=0xc000221928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc0005a44e0, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc0006eedc0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000221d70 sp=0xc000221950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc0006eedc0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000221f20 sp=0xc000221d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000221fe0 sp=0xc000221f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000221fe8 sp=0xc000221fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2823 [chan receive, 1209 minutes]:
runtime.gopark(0xb45ff8?, 0x1?, 0x40?, 0xe1?, 0xc000a1cb00?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc000c5f8b0 sp=0xc000c5f890 pc=0x43890e
runtime.chanrecv(0xc0002637a0, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000c5f928 sp=0xc000c5f8b0 pc=0x4060ad
runtime.chanrecv1(0xc000c5f998?, 0x4403d1?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000c5f950 sp=0xc000c5f928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc000283740, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000b882c0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000c5fd70 sp=0xc000c5f950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000b882c0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000c5ff20 sp=0xc000c5fd70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000c5ffe0 sp=0xc000c5ff20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000c5ffe8 sp=0xc000c5ffe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2838 [chan receive, 1209 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0006118b0 sp=0xc000611890 pc=0x43890e
runtime.chanrecv(0xc000262ba0, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000611928 sp=0xc0006118b0 pc=0x4060ad
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000611950 sp=0xc000611928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc00081fec0, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc0008ab8c0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000611d70 sp=0xc000611950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc0008ab8c0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000611f20 sp=0xc000611d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000611fe0 sp=0xc000611f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000611fe8 sp=0xc000611fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

goroutine 2839 [chan receive, 1209 minutes]:
runtime.gopark(0xb45ff8?, 0x1?, 0x40?, 0xe1?, 0xc000a1c700?)
        /usr/local/google/home/bcmills/go/src/runtime/proc.go:387 +0xce fp=0xc0002498b0 sp=0xc000249890 pc=0x43890e
runtime.chanrecv(0xc000263860, 0x0, 0x1)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:583 +0x3cd fp=0xc000249928 sp=0xc0002498b0 pc=0x4060ad
runtime.chanrecv1(0xc000249998?, 0x4403d1?)
        /usr/local/google/home/bcmills/go/src/runtime/chan.go:442 +0x12 fp=0xc000249950 sp=0xc000249928 pc=0x405cb2
cmd/go/internal/test.(*runTestActor).Act(0xc000283ec0, 0xece0c0?, {0xb4dd18, 0xece0c0}, 0xc000c426e0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/test/test.go:1214 +0x6a fp=0xc000249d70 sp=0xc000249950 pc=0x9365ea
cmd/go/internal/work.(*Builder).Do.func3({0xb4dd18, 0xece0c0}, 0xc000c426e0)
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:150 +0x7e4 fp=0xc000249f20 sp=0xc000249d70 pc=0x8d1be4
cmd/go/internal/work.(*Builder).Do.func4()
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:218 +0xad fp=0xc000249fe0 sp=0xc000249f20 pc=0x8d122d
runtime.goexit()
        /usr/local/google/home/bcmills/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000249fe8 sp=0xc000249fe0 pc=0x46a841
created by cmd/go/internal/work.(*Builder).Do in goroutine 1
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/work/exec.go:204 +0x3e8

rax    0xca
rbx    0x0
rcx    0x46c683
rdx    0x0
rdi    0xc000680548
rsi    0x80
rbp    0xc00068ddd8
rsp    0xc00068dd90
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x2c
r13    0xc0000120bc
r14    0xc000682680
r15    0x0
rip    0x46c681
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. GoCommand cmd/go labels May 15, 2023
@bcmills bcmills added this to the Backlog milestone May 15, 2023
@bcmills bcmills mentioned this issue May 16, 2023
Closed
@bcmills
Copy link
Contributor Author

bcmills commented May 16, 2023

I think this is a regression from CL 448357. It (*work.Builder).Do stops running actions as soon as the interrupt signal is received, but (*runTestActor).Act assumes that the Act invocation for the previous test will always occur.

@bcmills bcmills modified the milestones: Backlog, Go1.21 May 16, 2023
@twmb
Copy link
Contributor

twmb commented May 18, 2023 •
edited by bcmills

I've also been having deadlocks for a few months when ^Cing go test ./.... I haven't bothered to report it but, since this is open, I wonder if what I'm seeing is the same.

Here's some output after I ctrl+C:

^Ck8s.io/kube-openapi/pkg/schemaconv: /home/twmb/go/go1.20.3/pkg/tool/linux_amd64/vet: signal: interrupt
github.com/aws/aws-sdk-go/private/protocol/restjson: /home/twmb/go/go1.20.3/pkg/tool/linux_amd64/vet: signal: interrupt
github.com/aws/aws-sdk-go/aws/defaults: /home/twmb/go/go1.20.3/pkg/tool/linux_amd64/vet: signal: interrupt
github.com/redpanda-data/redpanda/src/go/rpk/pkg/cli/acl: /home/twmb/go/go1.20.3/pkg/tool/linux_amd64/vet: signal: interrupt
github.com/redpanda-data/redpanda/src/go/rpk/pkg/cli/acl.test: /home/twmb/go/go1.20.3/pkg/tool/linux_amd64/link: signal: interrupt
k8s.io/kube-openapi/pkg/spec3: /home/twmb/go/go1.20.3/pkg/tool/linux_amd64/vet: signal: interrupt
github.com/aws/aws-sdk-go/service/sts: /home/twmb/go/go1.20.3/pkg/tool/linux_amd64/vet: signal: interrupt
k8s.io/api/core/v1: /home/twmb/go/go1.20.3/pkg/tool/linux_amd64/vet: signal: interrupt

Here's some output of strace -p {pid} -f:

[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001cce000, 65536, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001cbe000, 65536, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001cae000, 65536, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001c9e000, 65536, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001c8e000, 65536, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001c80000, 57344, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001c74000, 8192, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001c64000, 65536, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001c54000, 65536, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001c44000, 65536, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001c34000, 65536, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001c24000, 65536, MADV_DONTNEED) = 0
[pid 469366] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469366] madvise(0xc001c14000, 65536, MADV_DONTNEED) = 0
[pid 469366] write(6, "\0", 1)          = 1
[pid 469371] <... epoll_pwait resumed>[{events=EPOLLIN, data={u32=15301064, u64=15301064}}], 128, -1, NULL, 0) = 1
[pid 469366] futex(0xc000062d48, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 469371] read(5, "\0", 16)          = 1
[pid 469371] epoll_pwait(4, [], 128, 0, NULL, 0) = 0
[pid 469371] epoll_pwait(4, [], 128, 2, NULL, 0) = 0
[pid 469371] epoll_pwait(4, [], 128, 0, NULL, 0) = 0
[pid 469371] epoll_pwait(4, [], 128, 1, NULL, 0) = 0
[pid 469371] futex(0xc000062d48, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 469366] <... futex resumed>)       = 0
[pid 469371] futex(0xc000580148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 469366] epoll_pwait(4,  <unfinished ...>
[pid 469371] <... futex resumed>)       = 1
[pid 469366] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 469369] <... futex resumed>)       = 0
[pid 469371] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 469366] epoll_pwait(4,  <unfinished ...>
[pid 469369] futex(0xc000580148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 469371] <... madvise resumed>)     = 0
[pid 469371] madvise(0xc001c04000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001c00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001c00000, 16384, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001bf2000, 49152, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001be2000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001bd2000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001bc2000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001bb2000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001b76000, 8192, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001b32000, 8192, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001b24000, 49152, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001b20000, 16384, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001b0a000, 49152, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001afe000, 49152, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001af6000, 16384, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001aee000, 16384, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001ae8000, 8192, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001ae4000, 8192, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001ad2000, 8192, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001ac6000, 8192, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001ac0000, 16384, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001ab0000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001aaa000, 24576, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001aa6000, 8192, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a90000, 32768, MADV_DONTNEED) = 0
[pid 469371] write(6, "\0", 1)          = 1
[pid 469366] <... epoll_pwait resumed>[{events=EPOLLIN, data={u32=15301064, u64=15301064}}], 128, -1, NULL, 0) = 1
[pid 469371] futex(0xc000126148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 469366] read(5, "\0", 16)          = 1
[pid 469366] epoll_pwait(4, [], 128, 0, NULL, 0) = 0
[pid 469366] epoll_pwait(4,  <unfinished ...>
[pid 469364] <... nanosleep resumed>NULL) = 0
[pid 469364] futex(0xe69680, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1756224} <unfinished ...>
[pid 469366] <... epoll_pwait resumed>[], 128, 2, NULL, 0) = 0
[pid 469366] epoll_pwait(4, [], 128, 0, NULL, 0) = 0
[pid 469366] epoll_pwait(4,  <unfinished ...>
[pid 469364] <... futex resumed>)       = -1 ETIMEDOUT (Connection timed out)
[pid 469364] nanosleep({tv_sec=0, tv_nsec=10000000},  <unfinished ...>
[pid 469366] <... epoll_pwait resumed>[], 128, 1, NULL, 0) = 0
[pid 469366] futex(0xc000126148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 469371] <... futex resumed>)       = 0
[pid 469366] <... futex resumed>)       = 1
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 469366] epoll_pwait(4,  <unfinished ...>
[pid 469371] <... madvise resumed>)     = 0
[pid 469366] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 469371] madvise(0xc001a80000, 65536, MADV_DONTNEED <unfinished ...>
[pid 469366] epoll_pwait(4,  <unfinished ...>
[pid 469371] <... madvise resumed>)     = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a7c000, 16384, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a74000, 8192, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a66000, 40960, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a56000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a46000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a36000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a26000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a20000, 24576, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a1c000, 8192, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a14000, 8192, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001a00000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001a0c000, 24576, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 4194304, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc0019fc000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc0019fa000, 8192, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc0019e4000, 57344, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc0019d4000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc0019d0000, 16384, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc0019c2000, 49152, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc0019b2000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc0019aa000, 32768, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc00199c000, 32768, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc00198a000, 32768, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001976000, 32768, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001966000, 65536, MADV_DONTNEED) = 0
[pid 469371] madvise(0xc001800000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 469371] madvise(0xc001956000, 65536, MADV_DONTNEED) = 0
[pid 469371] write(6, "\0", 1)          = 1
[pid 469366] <... epoll_pwait resumed>[{events=EPOLLIN, data={u32=15301064, u64=15301064}}], 128, -1, NULL, 0) = 1
[pid 469371] futex(0xc000126148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 469366] read(5, "\0", 16)          = 1
[pid 469366] epoll_pwait(4, [], 128, 0, NULL, 0) = 0
[pid 469366] epoll_pwait(4, [], 128, 2, NULL, 0) = 0
[pid 469366] epoll_pwait(4, [], 128, 0, NULL, 0) = 0
[pid 469366] epoll_pwait(4, [], 128, 1, NULL, 0) = 0
[pid 469366] futex(0xc000126148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 469371] <... futex resumed>)       = 0
[pid 469366] <... futex resumed>)       = 1
[pid 469371] epoll_pwait(4,  <unfinished ...>
[pid 469366] epoll_pwait(4,  <unfinished ...>
[pid 469371] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 469366] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 469371] epoll_pwait(4,  <unfinished ...>
[pid 469366] futex(0xc000062d48, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 469364] <... nanosleep resumed>NULL) = 0
[pid 469364] futex(0xe69680, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0}) = -1 ETIMEDOUT (Connection timed out)
[pid 469364] futex(0xc000062d48, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 469366] <... futex resumed>)       = 0
[pid 469364] nanosleep({tv_sec=0, tv_nsec=10000000},  <unfinished ...>
[pid 469366] epoll_pwait(4, [], 128, 0, NULL, 0) = 0
[pid 469366] futex(0xc000580148, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 469369] <... futex resumed>)       = 0
[pid 469369] futex(0xc00136a148, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 469413] <... futex resumed>)       = 0
[pid 469413] futex(0xc000062948, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 469365] <... futex resumed>)       = 0
[pid 469365] futex(0xc00136a548, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 469414] <... futex resumed>)       = 0
[pid 469365] <... futex resumed>)       = 1
[pid 469414] futex(0xc00136a548, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 469365] futex(0xc000062948, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 469413] futex(0xe69698, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000} <unfinished ...>
[pid 469366] futex(0xe69698, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 469369] sched_yield( <unfinished ...>
[pid 469366] <... futex resumed>)       = 0
[pid 469369] <... sched_yield resumed>) = 0
[pid 469366] futex(0xc000062d48, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 469413] <... futex resumed>)       = -1 EAGAIN (Resource temporarily unavailable)
[pid 469369] futex(0xe69598, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 469413] epoll_pwait(4,  <unfinished ...>
[pid 469369] <... futex resumed>)       = 0
[pid 469413] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 469369] futex(0xc000580148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 469413] futex(0xc000580148, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 469369] <... futex resumed>)       = 0
[pid 469413] futex(0xe69698, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000} <unfinished ...>
[pid 469369] futex(0xe69698, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 469413] <... futex resumed>)       = -1 EAGAIN (Resource temporarily unavailable)
[pid 469369] <... futex resumed>)       = 0
[pid 469369] nanosleep({tv_sec=0, tv_nsec=3000}, NULL) = 0
[pid 469369] futex(0xc000580148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 469413] futex(0xc00136a148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 469364] <... nanosleep resumed>NULL) = 0
[pid 469364] futex(0xe69680, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0}

Please let me know if this is related / if you'd like me to look into more.

@bcmills
Copy link
Contributor Author

bcmills commented May 18, 2023

@twmb, thanks for the report! Unfortunately there isn't a lot I can do to diagnose it from the strace output — sending SIGABRT to the main go process is what would give the stack traces we'd need. But since we have a good working theory for how it happens and when it started (#60220 reported that the bug was introduced in Go 1.20), I would expect that if you're on Go 1.20 or higher it's probably the same bug.

@bcmills bcmills added the NeedsFix The path to resolution is known, but the work has not been done. label May 31, 2023
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 31, 2023
@gopherbot
Copy link

Change https://go.dev/cl/503936 mentions this issue: cmd/go/internal/test: don't wait for previous test actions when interrupted

@bcmills bcmills self-assigned this Jun 15, 2023
@bcmills
Copy link
Contributor Author

bcmills commented Jun 16, 2023

@gopherbot, please backport to Go 1.20. This was a regression in Go 1.20, and causes annoying deadlocks (a UX issue) when attempting to cancel test runs.

@gopherbot gopherbot mentioned this issue Jun 16, 2023
Closed
@gopherbot
Copy link

Backport issue(s) opened: #60849 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@gopherbot
Copy link

Change https://go.dev/cl/504062 mentions this issue: [release-branch.go1.20] cmd/go/internal/test: don't wait for previous test actions when interrupted

gopherbot pushed a commit that referenced this issue Jun 22, 2023
@gopherbot
[release-branch.go1.20] cmd/go/internal/test: don't wait for previous…
b8e67d1 
… test actions when interrupted

Fixes #60849.
Updates #60203.

Change-Id: I59a3320ede1eb3cf4443d7ea37b8cb39d01f222a
Reviewed-on: https://go-review.googlesource.com/c/go/+/503936
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
(cherry picked from commit 6087671)
Reviewed-on: https://go-review.googlesource.com/c/go/+/504062
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
@bcmills bcmills mentioned this issue Sep 1, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bcmills bcmills

Labels
GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.21
Development

No branches or pull requests
3 participants
@twmb @bcmills @gopherbot