Joel Sing tells me that the old openbsd-386-rootbsd builder was MP, but openbsd-amd64-rootbsd builders was SP. Looking through old build.golang.org logs, I can't find any evidence of time.TimeAfterQueuing failing on openbsd-amd64-rootbsd (which aligns with my memories of thinking it was an openbsd/386-specific bug).

However, now openbsd-amd64-gce56 is MP too, and it's also exhibiting the bug. That seems to support my theory that it's a memory barrier issue.

Following up with OpenBSD to see about fixing kern_tc.c.

We should have a Github issue label for kernel-bugs-in-various-operating-systems-that-go-has-revealed

Still jet lagged, so I dug into this some more early this morning. On my X201s running OpenBSD -current, I was able to reproduce this issue. I think the kern_tc.c race is still technically a bug, but it doesn't seem to be the root cause of this crash.

My way of easily reproducing currently is:

sed -i.tmp 's/const attempts = 3/const attempts = 1/' src/time/sleep_test.go
go test -c time
yes | head -n 10000 | GOMAXPROCS=4 xargs -P20 -n1 ./time.test -test.short=true -test.timeout=120s -test.run=TestAfterQueuing | grep -v '^PASS$'

(Notably, so far I've only repro'd with GOMAXPROCS>1, but it seems like the builders are using GOMAXPROCS=1.)

A couple things to explain:

1. With -test.short=true, TestAfterQueuing lowers "delta" to 20ms.
2. On OpenBSD (at least the arches supported by Go), the processor scheduler has a period of 10ms, whereas on Linux they're 1ms. I.e., CPU intensive threads can potentially run for up to 10ms uninterrupted on OpenBSD.
3. When "go test std cmd" runs time.test, it's also busy compiling other packages/commands (i.e., CPU intensive work).

That leads to these consequences:

1. The first call to testAfterQueuing() tries to create a bunch of interleaved timers on lines 261--263. If the test process is preempted for >=20ms while creating the timers, it's possible (e.g.) the final After(0 * delta) call occurred more than delta after the call to After(1 * delta). I've seen this happen in ktrace/kdump output.

(Note: Subsequent retries, the global "slots" array is already sorted, so this cause goes away. To trigger the failure below with "attempts = 1", move the sort.Ints() call earlier.)

1. Even when the queued timers are for non-decreasing intervals, there's a possibility that the "timerproc" Go routine gets delayed for >=20ms. I've traced through timerproc to verify that even in this case, it processes the timers in the right order; but it's not guaranteed the "await" goroutines will be scheduled to execute in the same order.

(Interestingly, in a simplified test case, I'm able to reproduce this non-deterministic goroutine scheduling with GOMAXPROCS>1, but not with GOMAXPROCS=1. I need to investigate more to find out why this is.)

Takeaway: I don't think "kernel-bug" is an entirely accurate here, but because of OpenBSD's kernel behavior, it might make sense to only set "delta = 20 * Millisecond" for GOOS != "openbsd".

CL https://golang.org/cl/9510 mentions this issue.

Sigh, still flaking: http://build.golang.org/log/6ae361bdba9c95e35737fe173e291cd45959f1b3

(This was the build from 10f6d30 on build.golang.org for openbsd/386.)

CL https://golang.org/cl/9795 mentions this issue.