Looks like reproducing this with GOTRACEBACK=crash (or, do we already do that and it just didn't propagate to the test somehow?) will help; there are a bunch of goroutine running on other thread; stack unavailable goroutines that probably have some necessary information.

Perhaps related: the test sometimes fails on linux-amd64-longtest, e.g. https://build.golang.org/log/a86dc70291e4f1c534e8798731dcec91160e5e99:

##### maymorestack=mayMoreStackPreempt
--- FAIL: TestCgoPprofCallback (0.67s)
    testenv.go:468: [/workdir/tmp/go-build1237047364/testprogcgo.exe CgoPprofCallback] exit status: signal: segmentation fault (core dumped)
    crash_cgo_test.go:228: expected "OK\n" got 
FAIL
FAIL	runtime	72.817s

greplogs --dashboard -md -l -e 'CgoPprofCallback'

2022-09-06T11:14:09-1c50484/linux-amd64-longtest
2022-09-05T21:39:28-4c1ca42/linux-amd64-longtest
2022-09-05T08:28:34-bd5595d/linux-386-longtest
2022-09-05T08:12:37-67e6542/linux-amd64-longtest
2022-09-05T08:07:47-357b922/linux-amd64-longtest
2022-09-04T04:17:04-535fe2b/linux-amd64-longtest
2022-09-03T18:21:45-a0f0582/linux-386-longtest
2022-09-03T18:21:45-a0f0582/linux-amd64-longtest
2022-09-03T15:45:36-f798dc6/linux-amd64-longtest
2022-09-02T20:14:40-3e11e61/linux-amd64-longtest
2022-09-02T20:14:16-646c3ee/linux-amd64-longtest
2022-09-02T19:40:34-9154d4a/linux-amd64-longtest
2022-09-02T19:38:45-e406112/linux-amd64-longtest
2022-09-02T19:22:26-0fda8b1/linux-386-longtest
2022-09-02T19:08:56-b91e373/linux-amd64-longtest
2022-09-02T19:08:53-dbf442b/linux-amd64-longtest

I suspect this sudden influx is actually a new issue related to #54885, and that the ARM issue is not quite the same.

This is not the same as #54885, since that was introduced on 2022-09-02 and the original failure here happened well before that.

That one failure on linux/arm has been the only one since. Aside from a stream of persistent failures on longtest builders (that seem to be fixed), these are the latest 3:

2022-11-09T22:56:44-89332e0/openbsd-amd64-68
2022-11-01T14:20:31-ad5d2f6/linux-amd64-longtest
2022-11-01T14:11:54-2730170/openbsd-386-70

This is marked "release-blocker" but is also in "Backlog." I'm not sure what the means, so I'm moving it to the 1.20 milestone.

2022-11-09T22:56:44-89332e0/openbsd-amd64-68

This one appears to be unrelated.

One more failing TestCgoPprofCallback:
2022-12-09T21:38:33-e8f78cb/openbsd-amd64-68

This looks like a different bug, though.

This was discussed in the release weekly meeting-- it would be great to find out more about this bug so that we can re-evaluate it as a release blocker. Thanks.

The openbsd failures in particular could be the general corruption bugs tracked in #55161 / #34988.

Ping -- we're getting close to the release.

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "runtime" && test == "TestCgoPprofCallback"

--- FAIL: TestCgoPprofCallback (0.00s)
    crash_test.go:58: /tmp/workdir/tmp/go-build3123997662/testprogcgo.exe CgoPprofCallback failed to start: context deadline exceeded

--- FAIL: TestCgoPprofCallback (0.00s)
    crash_test.go:58: /tmp/workdir/tmp/go-build1527382388/testprogcgo.exe CgoPprofCallback failed to start: context deadline exceeded

--- FAIL: TestCgoPprofCallback (0.00s)
    crash_test.go:58: /tmp/workdir/tmp/go-build719402250/testprogcgo.exe CgoPprofCallback failed to start: context deadline exceeded

--- FAIL: TestCgoPprofCallback (0.03s)
    crash_test.go:55: building testprogcgo []: exit status 2
        fatal error: runtime: unblock on closing polldesc

        goroutine 335 [running]:
        runtime.throw({0xa52358?, 0x0?})
        	/tmp/workdir/go/src/runtime/panic.go:1047 +0x5d fp=0xc0005fbe38 sp=0xc0005fbe08 pc=0x43559d
        internal/poll.runtime_pollUnblock(0x227405bc8)
        	/tmp/workdir/go/src/runtime/netpoll.go:412 +0x22c fp=0xc0005fbe78 sp=0xc0005fbe38 pc=0x4622ec
        internal/poll.(*pollDesc).evict(...)
        	/tmp/workdir/go/src/internal/poll/fd_poll_runtime.go:61
        internal/poll.(*FD).Close(0xc0000f25a0)
        	/tmp/workdir/go/src/internal/poll/fd_unix.go:103 +0x45 fp=0xc0005fbea0 sp=0xc0005fbe78 pc=0x4d8c25
        os.(*file).close(0xc0000f25a0)
        	/tmp/workdir/go/src/os/file_unix.go:262 +0xad fp=0xc0005fbef8 sp=0xc0005fbea0 pc=0x4e73ed
        os.(*File).Close(...)
        	/tmp/workdir/go/src/os/file_posix.go:25
        os/exec.(*Cmd).writerDescriptor.func1()
        	/tmp/workdir/go/src/os/exec/exec.go:561 +0x57 fp=0xc0005fbf58 sp=0xc0005fbef8 pc=0x5143d7
        os/exec.(*Cmd).Start.func2(0x0?)
        	/tmp/workdir/go/src/os/exec/exec.go:717 +0x32 fp=0xc0005fbfc8 sp=0xc0005fbf58 pc=0x515112
        os/exec.(*Cmd).Start.func3()
        	/tmp/workdir/go/src/os/exec/exec.go:729 +0x2a fp=0xc0005fbfe0 sp=0xc0005fbfc8 pc=0x5150aa
        runtime.goexit()
        	/tmp/workdir/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005fbfe8 sp=0xc0005fbfe0 pc=0x467661
        created by os/exec.(*Cmd).Start
        	/tmp/workdir/go/src/os/exec/exec.go:716 +0xa33

— watchflakes

All the linux-amd64-longtest and linux-386-longtest failures are build with -dmaymorestack=mayMoreStackPreempt. They also haven't happened for some time.

All the other failures appear to be on slow machines. This is a CPU intensive test: it calls runtime.GOMAXPROCS(16) and then starts 64 goroutines. Each goroutine does an endless loop of sleeping for 50 microseconds and calling a Go function. The program keeps this up for a full second. And, the test is run in parallel with other tests.

So I think this is a flaky test. I think we should not return this CPU intensive test in parallel with other tests, and since it takes a full second I think we should not run it in short mode.

CC @prattmic for other opinions since he wrote the test.

Change https://go.dev/cl/460461 mentions this issue: runtime: skip TestCgoPprofCallback in short mode, don't run in parallel

The mayMoreStackPreempt ones are #54885, which is fixed.

The openbsd-.*-71 failures are probably #57585.