This looks like a fairly old bug, but the failure rate varies. It seems to have gone from “a couple per week” to “several per day” around 2021-11-04.

I wonder if this is related to #47629.

@4a6f656c, any ideas?

/tmp/workdir/go/src/runtime/sys_openbsd_386.s:524

That line intentionally faults when clock_gettime syscall fails. Why the syscall could fail? And why it is nondeterministic?

Investigations lead us to believe this is okay after Beta1 because this has been happening for some time, seems to be confined to this platform, and isn't a first-class port.

Just a shot in the dark, but does anyone know if openbsd's libc requires any particular alignment on SP? runtime·clock_gettime_trampoline doesn't do any alignment, so I wonder if that is tripping up a check in libc or the kernel.

I thought about that. I don't know. But we also don't align the SP in other syscall trampolines, whereas it always fails here. Also, if that's the case, I'd expect it fails much more deterministically.

I would have to double check, but I do not believe there are alignment requirements beyond those normally required by the hardware architecture itself (it is also trivial to write test code to verify this) - clock_gettime(2) should rarely/never fail and is only documented as returning EINVAL (invalid clock which is highly unlikely given how we call in) and EFAULT (bad memory). It is also worth noting that there is timekeep memory that can avoid libc from having to make an actual syscall, although if this was involved we'd probably be seeing faults within libc itself.

I suspect the underlying issue is some form of memory corruption given we're seeing other failures like faulting on unlock:

https://build.golang.org/log/3e594f9d7a015c79ab62629337b14a2e21a86e9d

Additionally, the openbsd-386-68 builder had been passing fairly consistently during the development cycle and things seem to have turned red around early November (although in limited testing I've not been able to trigger failures myself).

I would have to double check, but I do not believe there are alignment requirements beyond those normally required by the hardware architecture itself (it is also trivial to write test code to verify this) - clock_gettime(2) should rarely/never fail and is only documented as returning EINVAL (invalid clock which is highly unlikely given how we call in) and EFAULT (bad memory). It is also worth noting that there is timekeep memory that can avoid libc from having to make an actual syscall, although if this was involved we'd probably be seeing faults within libc itself.

I also clicked through the source in libc and the kernel and did not notice anything particularly noteworthy. No explicit alignment checks, no odd error paths beyond the ones @4a6f656c mentioned. The only thing of note is that it appeared that on 386, libc may really need to make a system call. I didn't see the fast path getting set up for 386 like it does for amd64, though I may well have missed it.

I would have to double check, but I do not believe there are alignment requirements beyond those normally required by the hardware architecture itself (it is also trivial to write test code to verify this) - clock_gettime(2) should rarely/never fail and is only documented as returning EINVAL (invalid clock which is highly unlikely given how we call in) and EFAULT (bad memory). It is also worth noting that there is timekeep memory that can avoid libc from having to make an actual syscall, although if this was involved we'd probably be seeing faults within libc itself.

I also clicked through the source in libc and the kernel and did not notice anything particularly noteworthy. No explicit alignment checks, no odd error paths beyond the ones @4a6f656c mentioned. The only thing of note is that it appeared that on 386, libc may really need to make a system call. I didn't see the fast path getting set up for 386 like it does for amd64, though I may well have missed it.

Ah, no you're correct - I neglected to recall that _tc_get_timecount is not set on OpenBSD i386, which means we're always calling the clock_gettime(2) syscall. That means that we're almost certainly failing due to EFAULT (ktrace would confirm if we can get a reproducible test case).

It seems the failure only occurs on openbsd-386-68 builder, not -70 or -70-n1 builders.

It is also interesting that all failures seem to come from the cmd/dist binary, not anything else.

Change https://golang.org/cl/368334 mentions this issue: runtime: print errno on clock_gettime failure on OpenBSD

CL 268334, which I just submitted, will change the message printed during this failure to a "clock_gettime failed" fatal error. New combined greplogs (not yet tested because there haven't been any new failures):

greplogs -dashboard -l -e "(?ms)\Aopenbsd-.*(clock_gettime_trampoline|fatal error: clock_gettime failed)"

It's possible I made this worse. I haven't seen any clock_gettime failures since my CL went in, but we did get a new "stopm holding p" with, unfortunately, no useful backtrace, which we haven't seen before.

Seems this hasn't occurred since @aclements 's CL...

For better or for worse, this is no longer happening, so closing.