The most recent ones which look like:

Building Go cmd/dist using /usr/home/paulzhol/go1.4. (go1.4-bootstrap-20170531 freebsd/arm)
Building Go toolchain1 using /usr/home/paulzhol/go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
go build cmd/compile/internal/ssa: /tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/tool/freebsd_arm/compile: signal: killed
go tool dist: FAILED: /tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/tool/freebsd_arm/go_bootstrap install -gcflags=all= -ldflags=all= -a -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1

Are due to out of memory during bootstrap. This also sometimes happened mid test like:
https://build.golang.org/log/04777783e4d4c1b95e6e9868f3d61e8bfaa076fe

They were due to the loss of a network connectivity to a block device used as swap, I've restored it. But it stil seems 1.18 is a much heavy memory user.

There used to be a way to control the bots by canceling and triggering a run a new. I think I've been locked out of that for several years. And besides it can never keep up with all the builds when it takes more than 3 houres per run.

it stil seems 1.18 is a much heavy memory user.

That could be due to #44167 — including non-heap sources of GC work in the pacing decisions can cause less frequent collection when those sources are a significant fraction of memory usage.

There used to be a way to control the bots by canceling and triggering a run a new. I think I've been locked out of that for several years.

The command that does that is golang.org/x/build/cmd/retrybuilds, although I'm not sure what permissions that requires. (Someone from @golang/release could probably fill in more detail.)

And besides it can never keep up with all the builds when it takes more than 3 [hours] per run.

I think that's fine in general, as long as the builder makes regular forward progress? (It seems ok to be missing test runs in the middle of a burst of changes as long as we occasionally get an up-to-date run at the end of the burst.)

I understand the Soon label was added to look into the sudden increase in failing builds starting with late December, and it seems it was determined to be related to an increase in memory use in Go 1.18. (If that's not working as expected, it probably needs a separate issue.) I'll remove the Soon label, since there doesn't appear to be a clear time-sensitive action that must be taken in the order of days or hours here.

There used to be a way to control the bots by canceling and triggering a run a new. I think I've been locked out of that for several years.

As Bryan mentioned, retrybuilds is that command. If you are the builder owner, I believe you should be able to use the builder key to clear out failures on that particular builder. It's expected there may be some changes happening as part of #47521, but please file an issue for that command if you can't use it on a builder you own.

It seems the current state of the builder is that it is missing. From https://farmer.golang.org/#pools:

host-freebsd-arm-paulzhol: 0/0 (1 missing)

greplogs --dashboard -md -l -e '(?ms)\Afreebsd-arm-paulzhol .* signal: killed\n' --since=2022-01-10

2022-04-23T05:38:56-9388b58-9717e8f/freebsd-arm-paulzhol
2022-03-27T20:26:46-3677212-0eea251/freebsd-arm-paulzhol
2022-03-24T21:51:49-a10a209/freebsd-arm-paulzhol
2022-03-11T21:51:20-baf61e4/freebsd-arm-paulzhol
2022-03-11T05:06:51-ef7d041/freebsd-arm-paulzhol
2022-02-28T22:24:58-936c7fb/freebsd-arm-paulzhol

2022-04-27T00:09 is the most recent. I'm not sure I can do anything about it.
RAM is very constrained, I already have a swap device mounted over ISCSI.
tmpfs is used as the filesystem for the build which should be swapped out under memory presure.

The path forward for FreeBSD on ARMv7 is probably running it virtualized under ARM64 anyway.

This is still happening frequently this week. I'm going to drop freebsd-arm-paulzhol from builder triage until it can be addressed.

I think I've found the root cause. I've managed to reproduce it by running all.bash on a rpi3b (more cores, less RAM, slower network):

The iscsid daemon (ISCSI initiatior control plane) plus the dhcient daemon were swapped out due to high memory presure. Later the lease can't be renewed and ISCSI can't re-establish the connection to the block device used for swap.

I've switched the following sysctls:

vm.swap_enabled=0
vm.pfault_oom_attempts=-1

The first one is meant to prevent whole process swapping (seperate from paging in FreeBSD) of runnable but inactive processes like iscsid.
While the second is to prevent a proccess from getting OOM Killed if paging cannot reclaim RAM fast enough (this was observed in several runs where dist got Killed during bootsrap).

Additionally I've moved /tmp to be mounted on a regualr UFS FS on an ISCSI block device, instead of tmpfs. This prevents heavy thrashing as the majority of the swapfile is used to hold the Go sources and build artifacts.