x/build: runtime package tests consistently fail due to 45 min timeout on darwin-amd64-longtest, only when running via all.bash #60919

dmitshur · 2023-06-21T14:57:26Z

Running all.bash on the darwin-amd64-longtest builder fairly consistently fails with a 45 minute timeout in the runtime package tests. Testing was done using at a near-tip commit on release-branch.go1.21 (ebbff91). From a sample of about 10 runs, there was one time the runtime package tests completed successfully:

ok  	runtime	2471.456s

But all other times they failed due to a 45 min timeout. (The 45 minutes comes from cmd/dist's default of 180 seconds for package tests, times 3 because cmd/go test are running, times 5 because the builder has GO_TEST_TIMEOUT_SCALE=5.)

panic: test timed out after 45m0s
running tests:
	TestSchedLocalQueueEmpty (21m8s)

[...]

FAIL	runtime	2700.568s

More

panic: test timed out after 45m0s
running tests:
	TestSchedLocalQueueEmpty (21m8s)

[...]

FAIL	runtime	2700.568s

--- FAIL: TestPingPongHog (0.19s)
    proc_test.go:490: want hogCount/lightCount in [0.05, 20]; got 8000000/1000 = 8000
panic: test timed out after 45m0s
running tests:
	TestSchedLocalQueueEmpty (20m31s)

[...]

FAIL	runtime	2700.967s

--- FAIL: TestPingPongHog (0.13s)
    proc_test.go:490: want hogCount/lightCount in [0.05, 20]; got 4000000/1000 = 4000
panic: test timed out after 45m0s
running tests:
	TestSchedLocalQueueEmpty (20m29s)

[...]

FAIL	runtime	2700.754s

FAIL	runtime	2701.087s
FAIL	runtime	2700.519s
FAIL	runtime	2700.616s
FAIL	runtime	2700.441s

Can be reproduced in a gomote with:

$ gomote create darwin-amd64-longtest
$ gomote puttar -dir gotip -url='https://go.googlesource.com/go/+archive/ebbff91f5930ba74a4285f8ba6ddfd0df0871d95.tar.gz' $USER-darwin-amd64-longtest-0
$ gomote ssh $USER-darwin-amd64-longtest-0
$ cd /tmp/buildlet/gotip/src
$ export GOROOT_BOOTSTRAP=$HOME/goboot
$ ./all.bash  # GO_TEST_SHORT=0 is implicitly on via this builder's environment

Here's a complete all.bash output from one such run:

gopher@Gophers-iMac-Pro src % time ./all.bash
Building Go cmd/dist using /Users/gopher/goboot. (go1.19.3 darwin/amd64)
Building Go toolchain1 using /Users/gopher/goboot.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.

##### Test execution environment.
# GOARCH: amd64
# CPU: 
# GOOS: darwin
# OS Version: Darwin 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct  9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64 x86_64

##### Testing packages.
ok  	archive/tar	0.684s
ok  	archive/zip	69.040s
ok  	bufio	0.226s
ok  	bytes	5.220s
ok  	cmp	0.079s
ok  	compress/bzip2	0.257s
ok  	compress/flate	20.785s
ok  	compress/gzip	6.121s
ok  	compress/lzw	0.290s
ok  	compress/zlib	2.880s
ok  	container/heap	0.099s
ok  	container/list	0.026s
ok  	container/ring	0.029s
ok  	context	0.168s
ok  	crypto	0.085s
ok  	crypto/aes	0.175s
ok  	crypto/cipher	0.800s
ok  	crypto/des	0.042s
ok  	crypto/dsa	23.812s
ok  	crypto/ecdh	1.521s
ok  	crypto/ecdsa	1.973s
ok  	crypto/ed25519	1.411s
ok  	crypto/elliptic	1.076s
ok  	crypto/hmac	0.033s
ok  	crypto/internal/alias	0.049s
ok  	crypto/internal/bigmod	0.550s
ok  	crypto/internal/boring	0.074s
ok  	crypto/internal/boring/bcache	0.275s
ok  	crypto/internal/edwards25519	10.263s
ok  	crypto/internal/edwards25519/field	7.944s
ok  	crypto/internal/nistec	12.088s
ok  	crypto/internal/nistec/fiat	0.037s [no tests to run]
ok  	crypto/md5	0.059s
ok  	crypto/rand	0.426s
ok  	crypto/rc4	0.068s
ok  	crypto/rsa	8.907s
ok  	crypto/sha1	0.054s
ok  	crypto/sha256	0.050s
ok  	crypto/sha512	0.052s
ok  	crypto/subtle	17.780s
ok  	crypto/tls	37.772s
ok  	crypto/x509	22.285s
ok  	database/sql	1.461s
ok  	database/sql/driver	0.029s
ok  	debug/buildinfo	33.520s
ok  	debug/dwarf	0.291s
ok  	debug/elf	1.157s
ok  	debug/gosym	31.489s
ok  	debug/macho	0.111s
ok  	debug/pe	0.291s
ok  	debug/plan9obj	0.054s
ok  	embed	0.097s [no tests to run]
ok  	embed/internal/embedtest	0.049s
ok  	encoding/ascii85	0.039s
ok  	encoding/asn1	0.054s
ok  	encoding/base32	0.094s
ok  	encoding/base64	0.035s
ok  	encoding/binary	0.037s
ok  	encoding/csv	0.036s
ok  	encoding/gob	11.098s
ok  	encoding/hex	0.046s
ok  	encoding/json	4.832s
ok  	encoding/pem	5.511s
ok  	encoding/xml	18.266s
ok  	errors	0.171s
ok  	expvar	0.070s
ok  	flag	0.223s
ok  	fmt	0.212s
ok  	go/ast	0.034s
ok  	go/build	2.346s
ok  	go/build/constraint	0.099s
ok  	go/constant	0.085s
ok  	go/doc	0.309s
ok  	go/doc/comment	3.958s
ok  	go/format	0.033s
ok  	go/importer	1.655s
ok  	go/internal/gccgoimporter	0.081s
ok  	go/internal/gcimporter	430.781s
ok  	go/internal/srcimporter	71.404s
ok  	go/parser	25.058s
ok  	go/printer	0.982s
ok  	go/scanner	0.096s
ok  	go/token	0.068s
ok  	go/types	84.196s
ok  	hash	0.046s
ok  	hash/adler32	0.048s
ok  	hash/crc32	0.055s
ok  	hash/crc64	0.043s
ok  	hash/fnv	0.052s
ok  	hash/maphash	300.247s
ok  	html	0.067s
ok  	html/template	1.890s
ok  	image	0.700s
ok  	image/color	0.052s
ok  	image/draw	0.289s
ok  	image/gif	41.522s
ok  	image/jpeg	13.865s
ok  	image/png	4.065s
ok  	index/suffixarray	114.748s
ok  	internal/abi	0.809s
ok  	internal/buildcfg	0.105s
ok  	internal/coverage/cformat	0.094s
ok  	internal/coverage/cmerge	0.033s
ok  	internal/coverage/pods	0.077s
ok  	internal/coverage/slicereader	0.035s
ok  	internal/coverage/slicewriter	0.051s
ok  	internal/coverage/test	0.055s
ok  	internal/cpu	0.184s
ok  	internal/dag	0.108s
ok  	internal/diff	0.088s
ok  	internal/fmtsort	0.186s
ok  	internal/fuzz	0.072s
ok  	internal/godebug	2.686s
ok  	internal/godebugs	0.068s
ok  	internal/intern	18.376s
ok  	internal/itoa	0.031s
ok  	internal/poll	0.944s
ok  	internal/profile	0.044s
ok  	internal/reflectlite	0.056s
ok  	internal/safefilepath	0.071s
ok  	internal/saferio	0.358s
ok  	internal/singleflight	0.053s
ok  	internal/testenv	1.993s
ok  	internal/trace	4.708s
ok  	internal/types/errors	2.962s
ok  	internal/unsafeheader	0.025s
ok  	internal/xcoff	0.049s
ok  	internal/zstd	0.036s
ok  	io	0.156s
ok  	io/fs	1.117s
ok  	io/ioutil	0.073s
ok  	log	0.054s
ok  	log/slog	0.554s
ok  	log/slog/internal/benchmarks	0.027s
ok  	log/slog/internal/buffer	0.129s
ok  	log/syslog	1.387s
ok  	maps	0.035s
ok  	math	0.035s
ok  	math/big	13.384s
ok  	math/bits	0.082s
ok  	math/cmplx	0.034s
ok  	math/rand	15.860s
ok  	mime	0.051s
ok  	mime/multipart	4.276s
ok  	mime/quotedprintable	6.464s
ok  	net	126.307s
ok  	net/http	153.486s
ok  	net/http/cgi	4.849s
ok  	net/http/cookiejar	0.101s
ok  	net/http/fcgi	0.164s
ok  	net/http/httptest	0.305s
ok  	net/http/httptrace	0.071s
ok  	net/http/httputil	23.016s
ok  	net/http/internal	0.059s
ok  	net/http/internal/ascii	0.028s
ok  	net/http/pprof	5.536s
ok  	net/internal/socktest	0.023s
ok  	net/mail	0.031s
ok  	net/netip	3.936s
ok  	net/rpc	0.362s
ok  	net/rpc/jsonrpc	0.112s
ok  	net/smtp	0.173s
ok  	net/textproto	3.713s
ok  	net/url	0.029s
ok  	os	19.573s
ok  	os/exec	4.130s
ok  	os/exec/internal/fdtest	0.054s
ok  	os/signal	17.377s
ok  	os/user	0.097s
ok  	path	0.059s
ok  	path/filepath	0.392s
ok  	plugin	0.032s
ok  	reflect	7.792s
ok  	regexp	59.010s
ok  	regexp/syntax	5.013s
--- FAIL: TestPingPongHog (0.21s)                                                                                                                              D     proc_test.go:490: want hogCount/lightCount in [0.05, 20]; got 6000000/1000 = 6000                                                                          ^ panic: test timed out after 45m0s
running tests:
	TestSchedLocalQueueEmpty (22m19s)

goroutine 73160259 [running]:
panic({0x13c5980?, 0xc000e86080?})
	/tmp/buildlet/gotip/src/runtime/panic.go:1017 +0x3ac fp=0xc000c32f10 sp=0xc000c32e60 pc=0x103b0ac
testing.(*M).startAlarm.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:2259 +0x3b9 fp=0xc000c32fe0 sp=0xc000c32f10 pc=0x1104059
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000c32fe8 sp=0xc000c32fe0 pc=0x10795c1
created by time.goFunc
	/tmp/buildlet/gotip/src/time/sleep.go:176 +0x2d

goroutine 1 [chan receive, 22 minutes]:
runtime.gopark(0xc0000cd970?, 0x10104c5?, 0x78?, 0x0?, 0x18?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc0000cd908 sp=0xc0000cd8e8 pc=0x103f0ee
runtime.chanrecv(0xc000192700, 0xc0000cd9ef, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc0000cd980 sp=0xc0000cd908 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc0000cd9a8 sp=0xc0000cd980 pc=0x1008f92
testing.(*T).Run(0xc0000de1a0, {0x145f7f5?, 0x20?}, 0x147ee20)
	/tmp/buildlet/gotip/src/testing/testing.go:1649 +0x3c8 fp=0xc0000cda68 sp=0xc0000cd9a8 pc=0x1100c68
testing.runTests.func1(0x1741240?)
	/tmp/buildlet/gotip/src/testing/testing.go:2054 +0x3e fp=0xc0000cdab8 sp=0xc0000cda68 pc=0x1102d7e
testing.tRunner(0xc0000de1a0, 0xc0000cdbd0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc0000cdb08 sp=0xc0000cdab8 pc=0x10ffe3f
testing.runTests(0xc00009a780?, {0x173c3a0, 0x1b7, 0x1b7}, {0x2ed000000000000?, 0xc0000cdc90?, 0x1740520?})
	/tmp/buildlet/gotip/src/testing/testing.go:2052 +0x445 fp=0xc0000cdc00 sp=0xc0000cdb08 pc=0x1102c65
testing.(*M).Run(0xc00009a780)
	/tmp/buildlet/gotip/src/testing/testing.go:1925 +0x636 fp=0xc0000cde48 sp=0xc0000cdc00 pc=0x1101656
runtime_test.TestMain(0x107923a?)
	/tmp/buildlet/gotip/src/runtime/crash_test.go:27 +0x13 fp=0xc0000cde88 sp=0xc0000cde48 pc=0x12ec513
main.main()
	_testmain.go:1543 +0x1c6 fp=0xc0000cdf40 sp=0xc0000cde88 pc=0x1399246
runtime.main()
	/tmp/buildlet/gotip/src/runtime/proc.go:267 +0x2bb fp=0xc0000cdfe0 sp=0xc0000cdf40 pc=0x103ec7b
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000cdfe8 sp=0xc0000cdfe0 pc=0x10795c1

goroutine 2 [force gc (idle), 36 minutes]:
runtime.gopark(0x2a11745527a?, 0x1741240?, 0x0?, 0x0?, 0x0?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc0000487a8 sp=0xc000048788 pc=0x103f0ee
runtime.goparkunlock(...)
	/tmp/buildlet/gotip/src/runtime/proc.go:404
runtime.forcegchelper()
	/tmp/buildlet/gotip/src/runtime/proc.go:322 +0xb3 fp=0xc0000487e0 sp=0xc0000487a8 pc=0x103ef53
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000487e8 sp=0xc0000487e0 pc=0x10795c1
created by runtime.init.6 in goroutine 1
	/tmp/buildlet/gotip/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x173ff01?, 0x173ffe0?, 0xc?, 0x14?, 0x1?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000048f78 sp=0xc000048f58 pc=0x103f0ee
runtime.goparkunlock(...)
	/tmp/buildlet/gotip/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
	/tmp/buildlet/gotip/src/runtime/mgcsweep.go:321 +0xdf fp=0xc000048fc8 sp=0xc000048f78 pc=0x1028b7f
runtime.gcenable.func1()
	/tmp/buildlet/gotip/src/runtime/mgc.go:200 +0x25 fp=0xc000048fe0 sp=0xc000048fc8 pc=0x101d725
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000048fe8 sp=0xc000048fe0 pc=0x10795c1
created by runtime.gcenable in goroutine 1
	/tmp/buildlet/gotip/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x2fc735?, 0x38561?, 0x0?, 0x0?, 0x0?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000049770 sp=0xc000049750 pc=0x103f0ee
runtime.goparkunlock(...)
	/tmp/buildlet/gotip/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x17405a0)
	/tmp/buildlet/gotip/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc0000497a0 sp=0xc000049770 pc=0x10263a9
runtime.bgscavenge(0x0?)
	/tmp/buildlet/gotip/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc0000497c8 sp=0xc0000497a0 pc=0x1026959
runtime.gcenable.func2()
	/tmp/buildlet/gotip/src/runtime/mgc.go:201 +0x25 fp=0xc0000497e0 sp=0xc0000497c8 pc=0x101d6c5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000497e8 sp=0xc0000497e0 pc=0x10795c1
created by runtime.gcenable in goroutine 1
	/tmp/buildlet/gotip/src/runtime/mgc.go:201 +0xa5

goroutine 5 [finalizer wait, 22 minutes]:
runtime.gopark(0x0?, 0x147f158?, 0x90?, 0xa3?, 0x2000000020?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000049e28 sp=0xc000049e08 pc=0x103f0ee
runtime.runfinq()
	/tmp/buildlet/gotip/src/runtime/mfinal.go:193 +0x107 fp=0xc000049fe0 sp=0xc000049e28 pc=0x101c727
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000049fe8 sp=0xc000049fe0 pc=0x10795c1
created by runtime.createfing in goroutine 1
	/tmp/buildlet/gotip/src/runtime/mfinal.go:163 +0x3d

goroutine 20213575 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x0?, 0x6e?, 0xc000d72c98?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000d72c48 sp=0xc000d72c28 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000d72cc0 sp=0xc000d72c48 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000d72ce8 sp=0xc000d72cc0 pc=0x1008f92
testing.(*T).Parallel(0xc0002681a0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000d72d80 sp=0xc000d72ce8 pc=0x10ffae5
runtime_test.TestCheckPtr(0xc0002681a0)
	/tmp/buildlet/gotip/src/runtime/checkptr_test.go:21 +0x8a fp=0xc000d72f70 sp=0xc000d72d80 pc=0x12e5f2a
testing.tRunner(0xc0002681a0, 0x147e5a8)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000d72fc0 sp=0xc000d72f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000d72fe0 sp=0xc000d72fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000d72fe8 sp=0xc000d72fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 35 [GC worker (idle), 23 minutes]:
runtime.gopark(0x3372d19298a?, 0xc000133e60?, 0x1a?, 0x14?, 0x0?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc00004a750 sp=0xc00004a730 pc=0x103f0ee
runtime.gcBgMarkWorker()
	/tmp/buildlet/gotip/src/runtime/mgc.go:1293 +0xe5 fp=0xc00004a7e0 sp=0xc00004a750 pc=0x101f2a5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00004a7e8 sp=0xc00004a7e0 pc=0x10795c1
created by runtime.gcBgMarkStartWorkers in goroutine 18
	/tmp/buildlet/gotip/src/runtime/mgc.go:1217 +0x1c

goroutine 34 [GC worker (idle), 22 minutes]:
runtime.gopark(0x344822df90a?, 0xc00002c180?, 0x1a?, 0x14?, 0x0?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000044f50 sp=0xc000044f30 pc=0x103f0ee
runtime.gcBgMarkWorker()
	/tmp/buildlet/gotip/src/runtime/mgc.go:1293 +0xe5 fp=0xc000044fe0 sp=0xc000044f50 pc=0x101f2a5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000044fe8 sp=0xc000044fe0 pc=0x10795c1
created by runtime.gcBgMarkStartWorkers in goroutine 18
	/tmp/buildlet/gotip/src/runtime/mgc.go:1217 +0x1c

goroutine 21 [GC worker (idle), 22 minutes]:
runtime.gopark(0x33e1446adec?, 0xc00002c1a0?, 0x1a?, 0x14?, 0x0?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000044750 sp=0xc000044730 pc=0x103f0ee
runtime.gcBgMarkWorker()
	/tmp/buildlet/gotip/src/runtime/mgc.go:1293 +0xe5 fp=0xc0000447e0 sp=0xc000044750 pc=0x101f2a5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000447e8 sp=0xc0000447e0 pc=0x10795c1
created by runtime.gcBgMarkStartWorkers in goroutine 18
	/tmp/buildlet/gotip/src/runtime/mgc.go:1217 +0x1c

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x344823bf331?, 0xc0001a0060?, 0x1a?, 0x14?, 0x0?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000045750 sp=0xc000045730 pc=0x103f0ee
runtime.gcBgMarkWorker()
	/tmp/buildlet/gotip/src/runtime/mgc.go:1293 +0xe5 fp=0xc0000457e0 sp=0xc000045750 pc=0x101f2a5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000457e8 sp=0xc0000457e0 pc=0x10795c1
created by runtime.gcBgMarkStartWorkers in goroutine 18
	/tmp/buildlet/gotip/src/runtime/mgc.go:1217 +0x1c

goroutine 36 [GC worker (idle), 22 minutes]:
runtime.gopark(0x33e1446a945?, 0x2?, 0x56?, 0xaa?, 0x0?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000498750 sp=0xc000498730 pc=0x103f0ee
runtime.gcBgMarkWorker()
	/tmp/buildlet/gotip/src/runtime/mgc.go:1293 +0xe5 fp=0xc0004987e0 sp=0xc000498750 pc=0x101f2a5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004987e8 sp=0xc0004987e0 pc=0x10795c1
created by runtime.gcBgMarkStartWorkers in goroutine 18
	/tmp/buildlet/gotip/src/runtime/mgc.go:1217 +0x1c

goroutine 8 [GC worker (idle), 23 minutes]:
runtime.gopark(0x3372d195998?, 0xc000133e80?, 0x1a?, 0x14?, 0x0?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc00004af50 sp=0xc00004af30 pc=0x103f0ee
runtime.gcBgMarkWorker()
	/tmp/buildlet/gotip/src/runtime/mgc.go:1293 +0xe5 fp=0xc00004afe0 sp=0xc00004af50 pc=0x101f2a5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00004afe8 sp=0xc00004afe0 pc=0x10795c1
created by runtime.gcBgMarkStartWorkers in goroutine 18
	/tmp/buildlet/gotip/src/runtime/mgc.go:1217 +0x1c

goroutine 20214033 [chan receive, 26 minutes]:
runtime.gopark(0x0?, 0xc001ef84e0?, 0xf0?, 0x80?, 0xc000a9ae00?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000a9adb0 sp=0xc000a9ad90 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000a9ae28 sp=0xc000a9adb0 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000a9ae50 sp=0xc000a9ae28 pc=0x1008f92
testing.(*T).Parallel(0xc000c03d40)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000a9aee8 sp=0xc000a9ae50 pc=0x10ffae5
runtime_test.TestDeferKeepAlive(0xc000c03d40)
	/tmp/buildlet/gotip/src/runtime/mfinal_test.go:237 +0x72 fp=0xc000a9af70 sp=0xc000a9aee8 pc=0x132afd2
testing.tRunner(0xc000c03d40, 0x147e668)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000a9afc0 sp=0xc000a9af70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000a9afe0 sp=0xc000a9afc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000a9afe8 sp=0xc000a9afe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213613 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x50?, 0x25?, 0xc000047dd0?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000047d80 sp=0xc000047d60 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000047df8 sp=0xc000047d80 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000047e20 sp=0xc000047df8 pc=0x1008f92
testing.(*T).Parallel(0xc000c031e0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000047eb8 sp=0xc000047e20 pc=0x10ffae5
runtime_test.TestCatchPanic(0xc000c031e0)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:457 +0x27 fp=0xc000047f70 sp=0xc000047eb8 pc=0x12ea1a7
testing.tRunner(0xc000c031e0, 0x147e498)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000047fc0 sp=0xc000047f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000047fe0 sp=0xc000047fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20205441 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x70?, 0x20?, 0xc000494e10?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000494dc0 sp=0xc000494da0 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000494e38 sp=0xc000494dc0 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000494e60 sp=0xc000494e38 pc=0x1008f92
testing.(*T).Parallel(0xc000c02000)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000494ef8 sp=0xc000494e60 pc=0x10ffae5
runtime_test.TestCgoCrashTraceback(0xc000c02000)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:240 +0x1c fp=0xc000494f70 sp=0xc000494ef8 pc=0x12e84bc
testing.tRunner(0xc000c02000, 0x147e4c0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000494fc0 sp=0xc000494f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000494fe0 sp=0xc000494fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000494fe8 sp=0xc000494fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213624 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0xd0?, 0x2a?, 0xc0004cf618?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc0004cf5c8 sp=0xc0004cf5a8 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc0004cf640 sp=0xc0004cf5c8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc0004cf668 sp=0xc0004cf640 pc=0x1008f92
testing.(*T).Parallel(0xc000c54340)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc0004cf700 sp=0xc0004cf668 pc=0x10ffae5
runtime_test.TestAbortInCgo(0xc000c54340)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:702 +0x1c fp=0xc0004cf770 sp=0xc0004cf700 pc=0x12eb65c
testing.tRunner(0xc000c54340, 0x147e380)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc0004cf7c0 sp=0xc0004cf770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc0004cf7e0 sp=0xc0004cf7c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004cf7e8 sp=0xc0004cf7e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213633 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc001ef82a0?, 0xd0?, 0x2d?, 0xc0004cfe08?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc0004cfdb8 sp=0xc0004cfd98 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc0004cfe30 sp=0xc0004cfdb8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc0004cfe58 sp=0xc0004cfe30 pc=0x1008f92
testing.(*T).Parallel(0xc000c54b60)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc0004cfef0 sp=0xc0004cfe58 pc=0x10ffae5
runtime_test.TestDestructorCallback(0xc000c54b60)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:809 +0x1f fp=0xc0004cff70 sp=0xc0004cfef0 pc=0x12ebf7f
testing.tRunner(0xc000c54b60, 0x147e6a0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc0004cffc0 sp=0xc0004cff70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc0004cffe0 sp=0xc0004cffc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004cffe8 sp=0xc0004cffe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213617 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0xe0?, 0x26?, 0xc000499e18?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000499dc8 sp=0xc000499da8 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000499e40 sp=0xc000499dc8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000499e68 sp=0xc000499e40 pc=0x1008f92
testing.(*T).Parallel(0xc000c03860)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000499f00 sp=0xc000499e68 pc=0x10ffae5
runtime_test.TestCgoTracebackSigpanic(0xc000c03860)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:537 +0x1c fp=0xc000499f70 sp=0xc000499f00 pc=0x12ea7bc
testing.tRunner(0xc000c03860, 0x147e580)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000499fc0 sp=0xc000499f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000499fe0 sp=0xc000499fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000499fe8 sp=0xc000499fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20214051 [chan receive, 26 minutes]:
runtime.gopark(0x0?, 0xc001ef84e0?, 0xf0?, 0x81?, 0xc000c30750?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000c30700 sp=0xc000c306e0 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000c30778 sp=0xc000c30700 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000c307a0 sp=0xc000c30778 pc=0x1008f92
testing.(*T).Parallel(0xc000c541a0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000c30838 sp=0xc000c307a0 pc=0x10ffae5
runtime_test.TestGcPacer(0xc000c541a0?)
	/tmp/buildlet/gotip/src/runtime/mgcpacer_test.go:17 +0x2a fp=0xc000c30f70 sp=0xc000c30838 pc=0x132c84a
testing.tRunner(0xc000c541a0, 0x147e7d0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000c30fc0 sp=0xc000c30f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000c30fe0 sp=0xc000c30fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000c30fe8 sp=0xc000c30fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213587 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x90?, 0xed?, 0xc000187608?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc0001875b8 sp=0xc000187598 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000187630 sp=0xc0001875b8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000187658 sp=0xc000187630 pc=0x1008f92
testing.(*T).Parallel(0xc00015f1e0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc0001876f0 sp=0xc000187658 pc=0x10ffae5
runtime_test.TestCgoPanicDeadlock(0xc00015f1e0)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:206 +0x1f fp=0xc000187770 sp=0xc0001876f0 pc=0x12e80df
testing.tRunner(0xc00015f1e0, 0x147e510)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc0001877c0 sp=0xc000187770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc0001877e0 sp=0xc0001877c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001877e8 sp=0xc0001877e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213972 [chan receive, 26 minutes]:
runtime.gopark(0x0?, 0xc001ef84e0?, 0x40?, 0x65?, 0xc0002d2e48?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc0002d2df8 sp=0xc0002d2dd8 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc0002d2e70 sp=0xc0002d2df8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc0002d2e98 sp=0xc0002d2e70 pc=0x1008f92
testing.(*T).Parallel(0xc00030a340)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc0002d2f30 sp=0xc0002d2e98 pc=0x10ffae5
runtime_test.TestMemmoveOverlapLarge0x120000(0xc00030a340)
	/tmp/buildlet/gotip/src/runtime/memmove_test.go:116 +0xa5 fp=0xc0002d2f70 sp=0xc0002d2f30 pc=0x131d0e5
testing.tRunner(0xc00030a340, 0x147ea70)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc0002d2fc0 sp=0xc0002d2f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc0002d2fe0 sp=0xc0002d2fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0002d2fe8 sp=0xc0002d2fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213606 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x50?, 0x22?, 0xc000c37c58?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000c37c08 sp=0xc000c37be8 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000c37c80 sp=0xc000c37c08 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000c37ca8 sp=0xc000c37c80 pc=0x1008f92
testing.(*T).Parallel(0xc000c02820)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000c37d40 sp=0xc000c37ca8 pc=0x10ffae5
runtime_test.testCgoPprof(0xc000c02820, {0x1456e8e, 0xe}, {0x1453579, 0x8}, {0x1452827, 0x6}, {0x1455b14, 0xc})
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:295 +0x79 fp=0xc000c37f18 sp=0xc000c37d40 pc=0x12e8ab9
runtime_test.TestCgoPprofPIE(0xc000a76fa0?)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:362 +0x45 fp=0xc000c37f70 sp=0xc000c37f18 pc=0x12e97e5
testing.tRunner(0xc000c02820, 0x147e528)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000c37fc0 sp=0xc000c37f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000c37fe0 sp=0xc000c37fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000c37fe8 sp=0xc000c37fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20212589 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x40?, 0xeb?, 0xc000a93df8?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000a93da8 sp=0xc000a93d88 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000a93e20 sp=0xc000a93da8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000a93e48 sp=0xc000a93e20 pc=0x1008f92
testing.(*T).Parallel(0xc00015e820)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000a93ee0 sp=0xc000a93e48 pc=0x10ffae5
runtime_test.TestCgoExternalThreadSIGPROF(0xc00015e820)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:92 +0x25 fp=0xc000a93f70 sp=0xc000a93ee0 pc=0x12e7585
testing.tRunner(0xc00015e820, 0x147e4e8)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000a93fc0 sp=0xc000a93f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000a93fe0 sp=0xc000a93fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000a93fe8 sp=0xc000a93fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213836 [chan receive, 26 minutes]:
runtime.gopark(0x0?, 0xc001ef84e0?, 0x90?, 0x20?, 0xc0004ce5e0?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc0004ce590 sp=0xc0004ce570 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc0004ce608 sp=0xc0004ce590 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc0004ce630 sp=0xc0004ce608 pc=0x1008f92
testing.(*T).Parallel(0xc000c02d00)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc0004ce6c8 sp=0xc0004ce630 pc=0x10ffae5
runtime_test.testConcurrentReadsAfterGrowth(0xc0004ce760?, 0x0)
	/tmp/buildlet/gotip/src/runtime/map_test.go:317 +0x2a fp=0xc0004ce750 sp=0xc0004ce6c8 pc=0x13145aa
runtime_test.TestConcurrentReadsAfterGrowth(0xc001711800?)
	/tmp/buildlet/gotip/src/runtime/map_test.go:363 +0x15 fp=0xc0004ce770 sp=0xc0004ce750 pc=0x1314d35
testing.tRunner(0xc000c02d00, 0x147e5f0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc0004ce7c0 sp=0xc0004ce770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc0004ce7e0 sp=0xc0004ce7c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004ce7e8 sp=0xc0004ce7e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213588 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0xf0?, 0xed?, 0xc000496e08?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000496db8 sp=0xc000496d98 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000496e30 sp=0xc000496db8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000496e58 sp=0xc000496e30 pc=0x1008f92
testing.(*T).Parallel(0xc00015f380)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000496ef0 sp=0xc000496e58 pc=0x10ffae5
runtime_test.TestCgoCCodeSIGPROF(0xc00015f380)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:216 +0x1f fp=0xc000496f70 sp=0xc000496ef0 pc=0x12e81ff
testing.tRunner(0xc00015f380, 0x147e4a0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000496fc0 sp=0xc000496f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000496fe0 sp=0xc000496fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000496fe8 sp=0xc000496fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213577 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0xd0?, 0x6e?, 0xc000d29660?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000d29610 sp=0xc000d295f0 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000d29688 sp=0xc000d29610 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000d296b0 sp=0xc000d29688 pc=0x1008f92
testing.(*T).Parallel(0xc0002684e0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000d29748 sp=0xc000d296b0 pc=0x10ffae5
runtime_test.TestCgoCrashHandler(0xc000d297a0?)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:24 +0x18 fp=0xc000d29770 sp=0xc000d29748 pc=0x12e6ff8
testing.tRunner(0xc0002684e0, 0x147e4b8)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000d297c0 sp=0xc000d29770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000d297e0 sp=0xc000d297c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000d297e8 sp=0xc000d297e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20212592 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x70?, 0xec?, 0xc000a935f8?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000a935a8 sp=0xc000a93588 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000a93620 sp=0xc000a935a8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000a93648 sp=0xc000a93620 pc=0x1008f92
testing.(*T).Parallel(0xc00015ed00)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000a936e0 sp=0xc000a93648 pc=0x10ffae5
runtime_test.TestCgoExecSignalMask(0xc00015ed00)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:135 +0x25 fp=0xc000a93770 sp=0xc000a936e0 pc=0x12e7925
testing.tRunner(0xc00015ed00, 0x147e4d8)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000a937c0 sp=0xc000a93770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000a937e0 sp=0xc000a937c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000a937e8 sp=0xc000a937e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213586 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x30?, 0xed?, 0xc000deed90?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000deed40 sp=0xc000deed20 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000deedb8 sp=0xc000deed40 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000deede0 sp=0xc000deedb8 pc=0x1008f92
testing.(*T).Parallel(0xc00015f040)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000deee78 sp=0xc000deede0 pc=0x10ffae5
runtime_test.TestCgoCheckBytes(0xc00015f040)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:166 +0x27 fp=0xc000deef70 sp=0xc000deee78 pc=0x12e7b87
testing.tRunner(0xc00015f040, 0x147e4b0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000deefc0 sp=0xc000deef70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000deefe0 sp=0xc000deefc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000deefe8 sp=0xc000deefe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213576 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x60?, 0x6e?, 0xc000de85e0?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000de8590 sp=0xc000de8570 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000de8608 sp=0xc000de8590 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000de8630 sp=0xc000de8608 pc=0x1008f92
testing.(*T).Parallel(0xc000268340)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000de86c8 sp=0xc000de8630 pc=0x10ffae5
runtime_test.TestCheckPtr2(0xc000268340)
	/tmp/buildlet/gotip/src/runtime/checkptr_test.go:74 +0x87 fp=0xc000de8770 sp=0xc000de86c8 pc=0x12e6447
testing.tRunner(0xc000268340, 0x147e5b0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000de87c0 sp=0xc000de8770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000de87e0 sp=0xc000de87c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000de87e8 sp=0xc000de87e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20212588 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0xe0?, 0xea?, 0xc000a90e08?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000a90db8 sp=0xc000a90d98 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000a90e30 sp=0xc000a90db8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000a90e58 sp=0xc000a90e30 pc=0x1008f92
testing.(*T).Parallel(0xc00015e680)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000a90ef0 sp=0xc000a90e58 pc=0x10ffae5
runtime_test.TestCgoExternalThreadPanic(0xc00015e680)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:80 +0x1f fp=0xc000a90f70 sp=0xc000a90ef0 pc=0x12e747f
testing.tRunner(0xc00015e680, 0x147e4e0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000a90fc0 sp=0xc000a90f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000a90fe0 sp=0xc000a90fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000a90fe8 sp=0xc000a90fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213651 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc001ef82a0?, 0x90?, 0x2e?, 0xc000a77e08?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000a77db8 sp=0xc000a77d98 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000a77e30 sp=0xc000a77db8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000a77e58 sp=0xc000a77e30 pc=0x1008f92
testing.(*T).Parallel(0xc000c54ea0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000a77ef0 sp=0xc000a77e58 pc=0x10ffae5
runtime_test.TestEnsureBindM(0xc000c54ea0)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:845 +0x1f fp=0xc000a77f70 sp=0xc000a77ef0 pc=0x12ec3ff
testing.tRunner(0xc000c54ea0, 0x147e6e0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000a77fc0 sp=0xc000a77f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000a77fe0 sp=0xc000a77fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000a77fe8 sp=0xc000a77fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213616 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x80?, 0x26?, 0xc000495608?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc0004955b8 sp=0xc000495598 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000495630 sp=0xc0004955b8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000495658 sp=0xc000495630 pc=0x1008f92
testing.(*T).Parallel(0xc000c036c0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc0004956f0 sp=0xc000495658 pc=0x10ffae5
runtime_test.TestSigStackSwapping(0xc000c036c0)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:517 +0x1f fp=0xc000495770 sp=0xc0004956f0 pc=0x12ea69f
testing.tRunner(0xc000c036c0, 0x147ee98)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc0004957c0 sp=0xc000495770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc0004957e0 sp=0xc0004957c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004957e8 sp=0xc0004957e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213625 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x30?, 0x2b?, 0xc0004d0608?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc0004d05b8 sp=0xc0004d0598 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc0004d0630 sp=0xc0004d05b8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc0004d0658 sp=0xc0004d0630 pc=0x1008f92
testing.(*T).Parallel(0xc000c544e0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc0004d06f0 sp=0xc0004d0658 pc=0x10ffae5
runtime_test.TestEINTR(0xc000c544e0)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:735 +0x1f fp=0xc0004d0770 sp=0xc0004d06f0 pc=0x12eb7df
testing.tRunner(0xc000c544e0, 0x147e6c0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc0004d07c0 sp=0xc0004d0770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc0004d07e0 sp=0xc0004d07c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004d07e8 sp=0xc0004d07e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213711 [chan receive, 41 minutes]:
runtime.gopark(0x0?, 0xc001ef84e0?, 0xb0?, 0x62?, 0xc000a72608?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000a725b8 sp=0xc000a72598 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000a72630 sp=0xc000a725b8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000a72658 sp=0xc000a72630 pc=0x1008f92
testing.(*T).Parallel(0xc000c54820)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000a726f0 sp=0xc000a72658 pc=0x10ffae5
runtime_test.TestRecoverBeforePanicAfterGoexit(0xc000c54820)
	/tmp/buildlet/gotip/src/runtime/crash_test.go:455 +0x38 fp=0xc000a72770 sp=0xc000a726f0 pc=0x12ef578
testing.tRunner(0xc000c54820, 0x147ed98)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000a727c0 sp=0xc000a72770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000a727e0 sp=0xc000a727c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000a727e8 sp=0xc000a727e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213971 [chan receive, 26 minutes]:
runtime.gopark(0x0?, 0xc001ef84e0?, 0xe0?, 0x64?, 0xc000a69e48?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000a69df8 sp=0xc000a69dd8 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000a69e70 sp=0xc000a69df8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000a69e98 sp=0xc000a69e70 pc=0x1008f92
testing.(*T).Parallel(0xc00030a1a0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000a69f30 sp=0xc000a69e98 pc=0x10ffae5
runtime_test.TestMemmoveLarge0x180000(0xc00030a1a0)
	/tmp/buildlet/gotip/src/runtime/memmove_test.go:104 +0xa5 fp=0xc000a69f70 sp=0xc000a69f30 pc=0x131cfe5
testing.tRunner(0xc00030a1a0, 0x147ea68)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000a69fc0 sp=0xc000a69f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000a69fe0 sp=0xc000a69fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000a69fe8 sp=0xc000a69fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213618 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x40?, 0x27?, 0xc0004d3618?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc0004d35c8 sp=0xc0004d35a8 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc0004d3640 sp=0xc0004d35c8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc0004d3668 sp=0xc0004d3640 pc=0x1008f92
testing.(*T).Parallel(0xc000c03a00)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc0004d3700 sp=0xc0004d3668 pc=0x10ffae5
runtime_test.TestCgoPanicCallback(0xc000c03a00)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:561 +0x1c fp=0xc0004d3770 sp=0xc0004d3700 pc=0x12ea9bc
testing.tRunner(0xc000c03a00, 0x147e508)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc0004d37c0 sp=0xc0004d3770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc0004d37e0 sp=0xc0004d37c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004d37e8 sp=0xc0004d37e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213605 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0xf0?, 0x21?, 0xc000849c58?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000849c08 sp=0xc000849be8 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000849c80 sp=0xc000849c08 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000849ca8 sp=0xc000849c80 pc=0x1008f92
testing.(*T).Parallel(0xc000c02680)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000849d40 sp=0xc000849ca8 pc=0x10ffae5
runtime_test.testCgoPprof(0xc000c02680, {0x0, 0x0}, {0x1453579, 0x8}, {0x1452827, 0x6}, {0x1455b14, 0xc})
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:295 +0x79 fp=0xc000849f18 sp=0xc000849d40 pc=0x12e8ab9
runtime_test.TestCgoPprof(0xc000047fa0?)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:358 +0x3d fp=0xc000849f70 sp=0xc000849f18 pc=0x12e977d
testing.tRunner(0xc000c02680, 0x147e518)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000849fc0 sp=0xc000849f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000849fe0 sp=0xc000849fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000849fe8 sp=0xc000849fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213652 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc001ef82a0?, 0xf0?, 0x2e?, 0xc00049b618?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc00049b5c8 sp=0xc00049b5a8 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc00049b640 sp=0xc00049b5c8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc00049b668 sp=0xc00049b640 pc=0x1008f92
testing.(*T).Parallel(0xc000c55040)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc00049b700 sp=0xc00049b668 pc=0x10ffae5
runtime_test.TestVDSO(0xc000c55040)
	/tmp/buildlet/gotip/src/runtime/crash_test.go:157 +0x1c fp=0xc00049b770 sp=0xc00049b700 pc=0x12ed8bc
testing.tRunner(0xc000c55040, 0x147f0f8)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc00049b7c0 sp=0xc00049b770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc00049b7e0 sp=0xc00049b7c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049b7e8 sp=0xc00049b7e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213844 [chan receive, 40 minutes]:
runtime.gopark(0x0?, 0xc001ef84e0?, 0x70?, 0x61?, 0xc000a9d638?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000a9d5e8 sp=0xc000a9d5c8 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000a9d660 sp=0xc000a9d5e8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000a9d688 sp=0xc000a9d660 pc=0x1008f92
testing.(*T).Parallel(0xc000268b60)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000a9d720 sp=0xc000a9d688 pc=0x10ffae5
runtime_test.TestGCTestMoveStackOnNextCall(0xc000268b60)
	/tmp/buildlet/gotip/src/runtime/gc_test.go:210 +0x1c fp=0xc000a9d770 sp=0xc000a9d720 pc=0x12fc01c
testing.tRunner(0xc000268b60, 0x147e798)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000a9d7c0 sp=0xc000a9d770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000a9d7e0 sp=0xc000a9d7c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000a9d7e8 sp=0xc000a9d7e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20212586 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x20?, 0xea?, 0xc0004d4608?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc0004d45b8 sp=0xc0004d4598 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc0004d4630 sp=0xc0004d45b8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc0004d4658 sp=0xc0004d4630 pc=0x1008f92
testing.(*T).Parallel(0xc00015e340)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc0004d46f0 sp=0xc0004d4658 pc=0x10ffae5
runtime_test.TestCgoTraceback(0xc00015e340)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:44 +0x1f fp=0xc0004d4770 sp=0xc0004d46f0 pc=0x12e719f
testing.tRunner(0xc00015e340, 0x147e560)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc0004d47c0 sp=0xc0004d4770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc0004d47e0 sp=0xc0004d47c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004d47e8 sp=0xc0004d47e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20212593 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0xd0?, 0xec?, 0xc000aa9e08?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000aa9db8 sp=0xc000aa9d98 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000aa9e30 sp=0xc000aa9db8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000aa9e58 sp=0xc000aa9e30 pc=0x1008f92
testing.(*T).Parallel(0xc00015eea0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000aa9ef0 sp=0xc000aa9e58 pc=0x10ffae5
runtime_test.TestEnsureDropM(0xc00015eea0)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:149 +0x1f fp=0xc000aa9f70 sp=0xc000aa9ef0 pc=0x12e7a5f
testing.tRunner(0xc00015eea0, 0x147e6e8)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000aa9fc0 sp=0xc000aa9f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000aa9fe0 sp=0xc000aa9fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000aa9fe8 sp=0xc000aa9fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

[...]

goroutine 20212587 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0x80?, 0xea?, 0xc000a96df8?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000a96da8 sp=0xc000a96d88 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000a96e20 sp=0xc000a96da8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000a96e48 sp=0xc000a96e20 pc=0x1008f92
testing.(*T).Parallel(0xc00015e4e0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000a96ee0 sp=0xc000a96e48 pc=0x10ffae5
runtime_test.TestCgoCallbackGC(0xc00015e4e0)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:53 +0x25 fp=0xc000a96f70 sp=0xc000a96ee0 pc=0x12e72c5
testing.tRunner(0xc00015e4e0, 0x147e4a8)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000a96fc0 sp=0xc000a96f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000a96fe0 sp=0xc000a96fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000a96fe8 sp=0xc000a96fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20213614 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0xb0?, 0x25?, 0xc000495e58?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000495e08 sp=0xc000495de8 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000495e80 sp=0xc000495e08 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000495ea8 sp=0xc000495e80 pc=0x1008f92
testing.(*T).Parallel(0xc000c03380)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000495f40 sp=0xc000495ea8 pc=0x10ffae5
runtime_test.TestCgoLockOSThreadExit(0xc000495fa0?)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:493 +0x18 fp=0xc000495f70 sp=0xc000495f40 pc=0x12ea4b8
testing.tRunner(0xc000c03380, 0x147e4f8)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000495fc0 sp=0xc000495f70 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000495fe0 sp=0xc000495fc0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000495fe8 sp=0xc000495fe0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad

goroutine 20212590 [chan receive, 43 minutes]:
runtime.gopark(0x0?, 0xc000076000?, 0xa0?, 0xeb?, 0xc000ded608?)
	/tmp/buildlet/gotip/src/runtime/proc.go:398 +0xce fp=0xc000ded5b8 sp=0xc000ded598 pc=0x103f0ee
runtime.chanrecv(0xc0000821e0, 0x0, 0x1)
	/tmp/buildlet/gotip/src/runtime/chan.go:583 +0x3cd fp=0xc000ded630 sp=0xc000ded5b8 pc=0x100938d
runtime.chanrecv1(0x173fe20?, 0x13c5980?)
	/tmp/buildlet/gotip/src/runtime/chan.go:442 +0x12 fp=0xc000ded658 sp=0xc000ded630 pc=0x1008f92
testing.(*T).Parallel(0xc00015e9c0)
	/tmp/buildlet/gotip/src/testing/testing.go:1403 +0x205 fp=0xc000ded6f0 sp=0xc000ded658 pc=0x10ffae5
runtime_test.TestCgoExternalThreadSignal(0xc00015e9c0)
	/tmp/buildlet/gotip/src/runtime/crash_cgo_test.go:106 +0x1f fp=0xc000ded770 sp=0xc000ded6f0 pc=0x12e76bf
testing.tRunner(0xc00015e9c0, 0x147e4f0)
	/tmp/buildlet/gotip/src/testing/testing.go:1595 +0xff fp=0xc000ded7c0 sp=0xc000ded770 pc=0x10ffe3f
testing.(*T).Run.func1()
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x25 fp=0xc000ded7e0 sp=0xc000ded7c0 pc=0x1100dc5
runtime.goexit()
	/tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000ded7e8 sp=0xc000ded7e0 pc=0x10795c1
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/gotip/src/testing/testing.go:1648 +0x3ad
FAIL	runtime	2700.374s
ok  	runtime/cgo	0.128s
ok  	runtime/coverage	219.696s
ok  	runtime/debug	0.538s
ok  	runtime/internal/atomic	2.918s
ok  	runtime/internal/math	0.035s
ok  	runtime/internal/sys	0.103s
ok  	runtime/internal/wasitest	0.037s
ok  	runtime/metrics	0.043s
ok  	runtime/pprof	172.532s
ok  	runtime/trace	24.523s
ok  	slices	0.529s
ok  	sort	1.569s
ok  	strconv	3.867s
ok  	strings	4.437s
ok  	sync	5.790s
ok  	sync/atomic	272.798s
ok  	syscall	5.363s
ok  	testing	6.910s
ok  	testing/fstest	0.043s
ok  	testing/iotest	0.023s
ok  	testing/quick	0.160s
ok  	testing/slogtest	0.035s
ok  	text/scanner	0.067s
ok  	text/tabwriter	0.022s
ok  	text/template	1.794s
ok  	text/template/parse	0.037s
ok  	time	16.225s
ok  	unicode	0.042s
ok  	unicode/utf16	0.054s
ok  	unicode/utf8	0.040s
ok  	cmd/addr2line	2.624s
ok  	cmd/api	55.041s
ok  	cmd/asm/internal/asm	2.978s
ok  	cmd/asm/internal/lex	0.026s
ok  	cmd/cgo/internal/swig	0.120s
ok  	cmd/cgo/internal/test	2.047s
ok  	cmd/cgo/internal/testcarchive	79.541s
ok  	cmd/cgo/internal/testcshared	33.894s
ok  	cmd/cgo/internal/testerrors	63.344s
ok  	cmd/cgo/internal/testfortran	0.200s
ok  	cmd/cgo/internal/testgodefs	7.664s
ok  	cmd/cgo/internal/testlife	5.389s
ok  	cmd/cgo/internal/testnocgo	0.023s
ok  	cmd/cgo/internal/testplugin	201.859s
ok  	cmd/cgo/internal/testsanitizers	0.027s [no tests to run]
ok  	cmd/cgo/internal/testshared	0.030s
ok  	cmd/cgo/internal/testso	8.682s
ok  	cmd/cgo/internal/teststdio	10.117s
ok  	cmd/cgo/internal/testtls	0.041s
ok  	cmd/compile/internal/abt	0.031s
ok  	cmd/compile/internal/amd64	27.447s
ok  	cmd/compile/internal/base	0.038s
ok  	cmd/compile/internal/compare	0.044s
ok  	cmd/compile/internal/dwarfgen	1.277s
ok  	cmd/compile/internal/importer	122.148s
ok  	cmd/compile/internal/ir	0.037s
ok  	cmd/compile/internal/logopt	1.958s
ok  	cmd/compile/internal/loopvar	133.111s
ok  	cmd/compile/internal/noder	0.040s
ok  	cmd/compile/internal/reflectdata	0.028s [no tests to run]
ok  	cmd/compile/internal/ssa	2044.246s
ok  	cmd/compile/internal/syntax	15.338s
ok  	cmd/compile/internal/test	272.817s
ok  	cmd/compile/internal/typecheck	1.585s
ok  	cmd/compile/internal/types	0.024s
ok  	cmd/compile/internal/types2	47.700s
ok  	cmd/covdata	4.944s
ok  	cmd/cover	4.858s
ok  	cmd/dist	0.052s
ok  	cmd/distpack	0.039s
ok  	cmd/doc	1.275s
ok  	cmd/fix	16.940s
ok  	cmd/go	2286.949s
ok  	cmd/go/internal/auth	0.107s
ok  	cmd/go/internal/cache	156.563s
ok  	cmd/go/internal/envcmd	0.288s
ok  	cmd/go/internal/fsys	0.300s
ok  	cmd/go/internal/generate	0.056s
ok  	cmd/go/internal/get	0.061s
ok  	cmd/go/internal/gover	0.041s
ok  	cmd/go/internal/imports	0.192s
ok  	cmd/go/internal/load	0.062s
ok  	cmd/go/internal/lockedfile	3.186s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.094s
ok  	cmd/go/internal/modconv	0.113s
ok  	cmd/go/internal/modfetch	88.177s
ok  	cmd/go/internal/modfetch/codehost	22.141s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.062s
ok  	cmd/go/internal/modindex	0.987s
ok  	cmd/go/internal/modload	132.943s
ok  	cmd/go/internal/mvs	0.043s
ok  	cmd/go/internal/par	0.048s
ok  	cmd/go/internal/str	0.038s
ok  	cmd/go/internal/test	0.142s
ok  	cmd/go/internal/toolchain	0.084s
ok  	cmd/go/internal/vcs	0.318s
ok  	cmd/go/internal/vcweb	0.113s
ok  	cmd/go/internal/vcweb/vcstest	25.225s
ok  	cmd/go/internal/web	0.055s
ok  	cmd/go/internal/work	1.410s
ok  	cmd/gofmt	78.050s
ok  	cmd/internal/archive	6.460s
ok  	cmd/internal/bootstrap_test	1321.297s
ok  	cmd/internal/buildid	2.248s
ok  	cmd/internal/cov	5.902s
ok  	cmd/internal/dwarf	0.035s
ok  	cmd/internal/edit	0.029s
ok  	cmd/internal/goobj	190.039s
ok  	cmd/internal/moddeps	415.238s
ok  	cmd/internal/notsha256	0.045s
ok  	cmd/internal/obj	4.979s
ok  	cmd/internal/obj/ppc64	22.048s
ok  	cmd/internal/obj/riscv	63.925s
ok  	cmd/internal/obj/s390x	0.024s
ok  	cmd/internal/obj/x86	18.306s
ok  	cmd/internal/objabi	0.037s
ok  	cmd/internal/pkgpath	0.293s
ok  	cmd/internal/pkgpattern	0.046s
ok  	cmd/internal/quoted	0.267s
ok  	cmd/internal/src	0.035s
ok  	cmd/internal/test2json	1.494s
ok  	cmd/link	128.266s
ok  	cmd/link/internal/benchmark	0.051s
ok  	cmd/link/internal/ld	196.680s
ok  	cmd/link/internal/loader	0.064s
ok  	cmd/nm	12.542s
ok  	cmd/objdump	23.895s
ok  	cmd/pack	7.383s
ok  	cmd/pprof	5.996s
ok  	cmd/trace	0.625s
ok  	cmd/vet	43.754s
FAIL
go tool dist: Failed: exit status 1
./all.bash  5245.55s user 10829.54s system 430% cpu 1:02:14.00 total
gopher@Gophers-iMac-Pro src %

Issue #59634 for x/tools looks related.

CC @golang/release, @prattmic.

The text was updated successfully, but these errors were encountered:

dmitshur · 2023-06-21T14:57:36Z

Some observations.

This doesn't happen when coordinator runs tests on build.golang.org. Runtime package tests report completing significantly faster on the very same underlying builder:
```
ok      runtime 356.053s
ok      runtime 560.500s
ok      runtime 618.895s
ok      runtime 618.291s
ok      runtime 597.883s
ok      runtime 483.716s
ok      runtime 498.443s
ok      runtime 565.885s
ok      runtime 365.870s
```
Notably, coordinator doesn't use all.bash directly; instead it takes responsibility for scheduling test runs itself (for purposes of sharding, but that doesn't apply here since there's only one shard). It seems it most commonly schedules the runtime test along with regexp/syntax and runtime/cgo tests:
```
run_tests_multi Gophers-iMac-Pro.local: [go_test:regexp/syntax go_test:runtime go_test:runtime/cgo]
```
This builder runs as a VM under QEMU, and the default GOMAXPROCS/NumCPU report 6. It's possible that causes all.bash to attempt more concurrency than is optimal.

I wanted to see if runtime package tests would pass successfully on the same builder if go test -timeout=50m cmd std were used used instead of all.bash, but they did not:

# (build toolchain, e.g., make.bash)
$ go test -timeout=50m std cmd && go tool dist test -run=: 
ok      archive/tar 0.710s
?       crypto/internal/boring/bbig [no test files]
[...]
panic: test timed out after 50m0s
running tests:
    TestGoroutineProfile (2m20s)

goroutine 51461419 [running]:
panic({0x13c5980?, 0xc001a34040?})
    /tmp/buildlet/gotip/src/runtime/panic.go:1017 +0x3ac fp=0xc001a58f10 sp=0xc001a58e60 pc=0x103b0ac
testing.(*M).startAlarm.func1()
    /tmp/buildlet/gotip/src/testing/testing.go:2259 +0x3b9 fp=0xc001a58fe0 sp=0xc001a58f10 pc=0x1104059
runtime.goexit()
    /tmp/buildlet/gotip/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc001a58fe8 sp=0xc001a58fe0 pc=0x10795c1
created by time.goFunc
    /tmp/buildlet/gotip/src/time/sleep.go:176 +0x2d

[...]
FAIL    runtime 3000.395s
[...]

I wanted to check if it would act differently than using all.bash, but it seems to reproduce the problem.

I remember talking with @aclements about further improvements to cmd/dist's scheduling of std cmd package tests as mentioned at https://cs.opensource.google/go/go/+/master:src/cmd/dist/test.go;l=920-925;drc=f4086d36862d8fd41af90356c894d04d3ff58203, which may be relevant.

bcmills · 2023-06-21T15:08:31Z

In #59634 our theory was that the builder was hitting swap (and may still be, for some tests).

Can we measure the low water mark of available memory on the builder during an all.bash run?
(Better still, can we record which test processes are running when the low water mark is hit?)

dmitshur · 2023-06-27T14:40:31Z

I noticed we use the same GO_TEST_TIMEOUT_SCALE value of 5 for this -longtest builder as for the other ones, which don't have as much performance overhead. Doubling it from 5 to 10 allowed all.bash to complete successfully twice in a row. (Their runtime package test timings were 2725.304s and 2821.530s, just a little over their previous 45 minute timeout.)

I checked the output of the memory_pressure command while the runtime tests were running, and it did not provide evidence of swap being used at all. It's possible the output was misleading and a better way of measuring would give more useful information.

Mailed CL 506515 to unblock bringing back this builder during release testing, where its speed is far from being a bottleneck. That said, improving the builder speed can still be done independently, but this issue was about it reproducibly failing to pass at all, which I think will be resolved.

gopherbot · 2023-06-27T14:41:00Z

Change https://go.dev/cl/504524 mentions this issue: internal/releasetargets: reapply "set LongTestBuilder for darwin-amd64"

gopherbot · 2023-06-27T14:41:01Z

Change https://go.dev/cl/506515 mentions this issue: dashboard: double darwin-amd64-longtest timeout scale

This reapplies CL 503758, which was rolled back in CL 503516. Trying again after scaling watchdog timeout to take into account that this is a longtest builder and it's expected to take longer, and after doubling its timeout scale value in the builder configuration. Since it's faster than signing, it's not expected to have any effect on the speed of the release process. We also have the option of using the "skip testing" workflow parameter if needing to temporarily skip it again, something I forgot about when rolling it back earlier. For golang/go#29252. For golang/go#60919. Change-Id: I179479b46a31ec5e95d4e9cd8b1f8fedb765880c Reviewed-on: https://go-review.googlesource.com/c/build/+/504524 Reviewed-by: Heschi Kreinick <heschi@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

dmitshur added this to the Unreleased milestone Jun 21, 2023

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 27, 2023

gopherbot closed this as completed in golang/build@d4cacbc Jun 27, 2023

dmitshur self-assigned this Jun 27, 2023

bcmills mentioned this issue Aug 22, 2023

x/build: darwin-amd64-longtest builds are slow #62215

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build: runtime package tests consistently fail due to 45 min timeout on darwin-amd64-longtest, only when running via all.bash #60919

x/build: runtime package tests consistently fail due to 45 min timeout on darwin-amd64-longtest, only when running via all.bash #60919

dmitshur commented Jun 21, 2023

dmitshur commented Jun 21, 2023 •

edited

bcmills commented Jun 21, 2023

dmitshur commented Jun 27, 2023 •

edited

gopherbot commented Jun 27, 2023

gopherbot commented Jun 27, 2023

x/build: runtime package tests consistently fail due to 45 min timeout on darwin-amd64-longtest, only when running via all.bash #60919

x/build: runtime package tests consistently fail due to 45 min timeout on darwin-amd64-longtest, only when running via all.bash #60919

Comments

dmitshur commented Jun 21, 2023

dmitshur commented Jun 21, 2023 • edited

bcmills commented Jun 21, 2023

dmitshur commented Jun 27, 2023 • edited

gopherbot commented Jun 27, 2023

gopherbot commented Jun 27, 2023

dmitshur commented Jun 21, 2023 •

edited

dmitshur commented Jun 27, 2023 •

edited