runtime: OpenBSD wirep related failures #49259

mknyszek · 2021-11-01T14:30:13Z

On trybots and in CI I've noticed a recent uptick in failures on OpenBSD, looking like

2021-11-01T02:47:30-fde4cc2/openbsd-amd64-68
2021-10-29T22:29:31-c812b97/openbsd-386-68
2021-10-28T03:44:12-2ff1074/openbsd-amd64-68

At first I thought this was one of my CLs, but the oldest one seems to indicate it's something else.

There are two other older ones that may be related:

2021-10-26T05:05:24-23fdd7f/openbsd-386-68
2021-10-26T01:22:47-adfb85b/openbsd-386-68

mknyszek · 2021-11-01T23:37:51Z

Trybot failure: https://storage.googleapis.com/go-build-log/6a1f8d0e/openbsd-amd64-68_5c29e56d.log

bcmills · 2021-11-02T16:33:54Z

The wirep failure mode may be new, but the OpenBSD runtime crashes are very much not. I suspect that this is a new manifestation of some existing source of memory corruption.

greplogs --dashboard -md -l -e '(?ms)\Aopenbsd-386.*fatal error:' --since=2021-06-01

bcmills · 2021-11-02T16:35:28Z

The openbsd-amd64 involvement may be new, though:

greplogs --dashboard -md -l -e '(?ms)\Aopenbsd-amd64.*fatal error:' --since=2021-06-01

2021-11-02T00:12:17-4ff95d6-81fea0b/openbsd-amd64-68
2021-11-01T02:47:30-fde4cc2/openbsd-amd64-68
2021-10-28T03:44:12-2ff1074/openbsd-amd64-68
2021-10-27T18:08:49-08f2e89-bb49eb3/openbsd-amd64-68

mknyszek · 2021-11-02T16:50:03Z

Yeah, I think the amd64 one is the new thing. It's also higher priority since that's a trybot and it's causing trybot flakes.

bcmills · 2021-11-03T15:56:55Z

TryBot: https://storage.googleapis.com/go-build-log/7121e9e5/openbsd-amd64-68_8fa9e26d.log

bcmills · 2021-11-03T16:03:21Z

This is a release blocker for 1.18 in that we need to figure out what to do about it.

OpenBSD is not a first-class port, so this does not necessarily block the 1.18 release — but if we are intentionally releasing 1.18 with unstable OpenBSD support its status should be documented in the release notes.

ALTree · 2021-11-06T12:19:48Z

trybot error (not sure if related): https://storage.googleapis.com/go-build-log/25f64550/openbsd-amd64-68_09e5b260.log

fatal error: bad g->status in ready

mknyszek · 2021-11-09T22:30:52Z

Spinning up gomotes for this too. Going to attempt at getting a core dump.

mknyszek · 2021-11-10T03:28:58Z

Took about 4 hours and 15 minutes of 10 gomotes spinning, but I reproduced it. No core file. :( I'd poke around the gomote but unfortunately I got to it too late. I'll try again tomorrow.

mknyszek · 2021-11-11T18:30:58Z

Wow. I've been spinning in 10 gomotes for 22h without a crash.

bcmills · 2021-11-11T19:54:10Z

Neato! Most (but not all) of the openbsd-386 crashes for the past day are in clock_gettime_trampoline. I see one unlock2, and the only amd64 failure is a new-looking netpoll failure mode.

greplogs --dashboard -md -l -e '(?ms)\Aopenbsd-.*fatal error:' --since=2021-11-10

2021-11-11T16:17:21-666fc17/openbsd-386-68
2021-11-11T04:02:33-3949faf/openbsd-386-68
2021-11-10T21:59:24-23f653d/openbsd-amd64-68
2021-11-10T21:53:03-229b909/openbsd-386-68
2021-11-10T18:06:32-097aaa9/openbsd-386-68
2021-11-10T17:15:54-8a3be15/openbsd-386-68
2021-11-10T05:08:25-17980df/openbsd-386-68
2021-11-10T02:26:41-02d7eab/openbsd-386-68
2021-11-10T00:45:37-ec86bb5/openbsd-386-68
2021-11-10T00:44:59-f5f9434/openbsd-386-68

bcmills · 2021-11-11T19:57:22Z

(Filed clock_gettime_trampoline separately as #49532, since it looks like it may be an older bug.)

aclements · 2021-12-01T17:32:46Z

I think we should focus this issue on the wirep failures, and use #49209 to track the other OpenBSD/NetBSD failures that are more obviously memory corruption. The wirep failures might also be that problem, but I'm not sure.

I think the wirep failures have two modes:

The runtime prints "wirep: invalid p state" and crashes.
The runtime prints "runtime: newstack at runtime.printlock" with a traceback from exitsyscallfast -> wirep. In this case, wirep is attempting to print "wirep: invalid p state" at a time when it can't call splittable functions like print.

Widening the net to both failures modes, we get

greplogs -dashboard -e "(?ms)^fatal error: wirep: invalid p state|^runtime: newstack.*^runtime.wirep" -l -md -since=2021-05-01

aclements · 2021-12-01T17:39:42Z

Actually, now I'm pretty confident this has the same root cause as #49209. Since 11-24, we've had several failures on the n2d (AMD) builders and none on the n2 (Intel) builders. And if we look at the two crashes that were able to print the invalid p state, in both cases p.m is 0:

greplogs -dashboard -e "(?ms)^fatal error: wirep: invalid p state" -l -md -since=2021-05-01
2021-11-18T16:18:50-feb330d/openbsd-amd64-68
2021-11-17T04:31:50-9c60a09/openbsd-amd64-68

Closing in favor of #49209.

mknyszek added the NeedsInvestigation label Nov 1, 2021

mknyszek added this to the Go1.18 milestone Nov 1, 2021

mknyszek added the OS-OpenBSD label Nov 1, 2021

mknyszek mentioned this issue Nov 2, 2021

runtime: pollCache.free crash #49273

Closed

bcmills added the release-blocker label Nov 3, 2021

bcmills mentioned this issue Nov 9, 2021

runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11 #49209

Closed

toothrot added the okay-after-beta1 label Nov 19, 2021

aclements closed this as completed Dec 1, 2021

thanm mentioned this issue Dec 9, 2021

Go compiler and runtime meeting notes #43930

Open

golang locked and limited conversation to collaborators Dec 1, 2022

gopherbot added the FrozenDueToAge label Dec 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: OpenBSD wirep related failures #49259

runtime: OpenBSD wirep related failures #49259

mknyszek commented Nov 1, 2021

mknyszek commented Nov 1, 2021

bcmills commented Nov 2, 2021

bcmills commented Nov 2, 2021

mknyszek commented Nov 2, 2021

bcmills commented Nov 3, 2021

bcmills commented Nov 3, 2021

ALTree commented Nov 6, 2021 •

edited

Loading

mknyszek commented Nov 9, 2021

mknyszek commented Nov 10, 2021

mknyszek commented Nov 11, 2021

bcmills commented Nov 11, 2021

bcmills commented Nov 11, 2021 •

edited

Loading

aclements commented Dec 1, 2021

aclements commented Dec 1, 2021

runtime: OpenBSD wirep related failures #49259

runtime: OpenBSD wirep related failures #49259

Comments

mknyszek commented Nov 1, 2021

mknyszek commented Nov 1, 2021

bcmills commented Nov 2, 2021

bcmills commented Nov 2, 2021

mknyszek commented Nov 2, 2021

bcmills commented Nov 3, 2021

bcmills commented Nov 3, 2021

ALTree commented Nov 6, 2021 • edited Loading

mknyszek commented Nov 9, 2021

mknyszek commented Nov 10, 2021

mknyszek commented Nov 11, 2021

bcmills commented Nov 11, 2021

bcmills commented Nov 11, 2021 • edited Loading

aclements commented Dec 1, 2021

aclements commented Dec 1, 2021

ALTree commented Nov 6, 2021 •

edited

Loading

bcmills commented Nov 11, 2021 •

edited

Loading