Yes, the disk (partition) was full. That particular machine was running the buildlet in a loop without re-booting between runs. I think it was the testcache which grew too much.

Given the overhead cost of saving test inputs in the cache (especially on Plan 9, but it's a nonzero cost anywhere), and given that every builder run should be independent and never use cached results, might it be a good idea to disable testcache for test runs on the builders? For the subrepo tests it's just a matter of adding -count 1 to the parameters of the go test command sent to the builder. For the main tree test, I'm not sure how to disable the testcache with go tool dist test, but there ought to be simple way.

Given that most of the Go project's own builders run in VMs, I would guess that the buildlet binary assumes that the build cache goes away completely in between runs.

Ah — here it is!
https://cs.opensource.google/go/x/build/+/master:cmd/buildlet/buildlet.go;l=211-212;drc=6d1c239cf4abb0e612c783ff67bf13966713df5f

For some reason the logic to clean the cache seems to be skipped on windows and plan9 builders, and I don't see any comments in the code or in CL 144637 explaining why. @dmitshur, @bradfitz: can you offer any insight?

The usual plan9-arm builders re-boot between runs with a fresh empty disk partition, so normally there wouldn't be a cache to clean; so that's not a problem. My suggestion to disable the testcache for builders is because of the time overhead in saving testcache input information (involving sometimes huge numbers of os.Stat calls) which will never be used.

greplogs -l -e '(?ms)\Aplan9-arm .* disk is full' --since=2022-01-28

(0 matching logs)