Given the very high failure rate (especially in November), and especially since linux/arm64 is a first class port, marking as release-blocker for Go 1.18 via #11811.

I suspect that this is an issue with either the builder configuration or the test itself (maybe this is an OOM failure?), so this does not need to block the Go 1.18 beta. (CC @golang/release)

This is the same issue we had here #44541. At the time @pmur submitted a CL to increase the memory limit. It must need to be increased again due to changes in the toolchain.
As noted in this previous issue, there is no limit on the ppc64 BE builder, so the error doesn't happen there.

/cc @heschi

Change https://golang.org/cl/370376 mentions this issue: cmd/api: run half as many go list calls in parallel

Seems it hasn't happened since the CL above. Can we call it fixed? Or we still wait for increasing the memory limit? Thanks.

Let's call it “probably fixed”. (We can always reopen if it isn't really.)

Unfortunately still occurring. I'll investigate.

greplogs --dashboard -md -l -e '##### API check\n(?:[^#].*\n)*.*signal: killed' --since=2021-12-09

2022-01-24T21:27:33-671e115/linux-ppc64le-power9osu
2022-01-24T21:27:25-7520c08/linux-ppc64le-buildlet

Change https://golang.org/cl/380994 mentions this issue: cmd/api: reduce parallel 'go list' invocations to a constant

😩

greplogs --dashboard -md -l -e '##### API check\n(?:[^#].*\n)*.*signal: killed' --since=2022-01-26

2022-03-23T19:31:11-515135f/linux-ppc64le-power9osu

greplogs --dashboard -md -l -e '##### API check\n(?:[^#].*\n)*.*signal: killed' --since=2022-03-30

2022-04-11T16:34:30-1930977/linux-ppc64le-buildlet
2022-04-11T16:31:40-6130b88/linux-ppc64le-buildlet
2022-04-11T16:31:36-7575811/linux-ppc64le-power9osu

Very odd how they're time-correlated like that, though. 🤔

@laboger, @pmur: is it possible that these builders are running on an overcommitted machine, or otherwise not fully memory-isolated from each other?

Also curious that the CPU: line in the header is blank (CC @prattmic).

##### Test execution environment.
# GOARCH: ppc64le
# CPU: 
# GOOS: linux
# OS Version: Linux 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54 UTC 2019 ppc64le

It would be nice to get more RAM in these hosts, but maybe in the meantime a simpler workaround would be to reduce to 4 builders, since that will use a total of 19.6 GiB of RAM.

See also #52105 . It seems the work is generally done in /workdir, which is tmpfs. If we move it to disk, it is probably less likely to OOM. I did that workaround for building the race detector syso.

I have opened a request with OSU to see if they can increase the RAM allocations on the PPC64 builders to 30GB.

Also curious that the CPU: line in the header is blank (CC @prattmic).

The CPU name ultimately comes from internal/cpu.Name, which is only implemented for 386 and amd64: https://cs.opensource.google/go/go/+/master:src/internal/cpu/cpu_no_name.go

No more of these within the last month or so. @pmur, was this fixed by a builder configuration change?

greplogs -l -e '##### API check\n(?:[^#].*\n)*.*signal: killed' --since=2022-04-12

(0 matching logs)

I poked OSU, and they increased RAM from 20 to 30GB on the VMs. That happened a few weeks ago. It probably helped.

Thanks! In that case, I'll consider this fixed unless / until we see otherwise. 👍