"Bryan C. Mills" ***@***.***> writes:

[...]

I've looked around and there may be several issues: * In every one of the failing builds, /tmp was full. It's ca. 40 GB no the build host, but resides in tmpfs, thus shares space with swap. * While golang left ca. 931 MB around in /tmp/workdir-host-solaris-oracle-amd64-oraclerel (almost half of that in go/pkg/obj/go-build) even after the build service was stopped, there still was plenty of free space left. * I don't the parallelism is too high: the golang buildlets uses 4 cores max, and an llvm buildbot running on the same host another 8, while the host has 24 cores. * Given all this, I suspect (but this is just a hunch) that some llvm testcase either exhausts /tmp (unlikely given that the llvm tmp files seem to reside in /var/tmp exclusively) or VM/swap (way more likely: I had runaway llvm testcases like this in the past), and if they are as lazy with resource control as they are with cleaning up tmp files (350k files in /var/tmp), this seems to be the most plausible cause. For remedy, there are several options: * Increasing either/or RAM or swap. * Limit the VM consumption of the services. I'll look into either of those.

"Bryan C. Mills" ***@***.***> writes:

[...] I was recently forced to migrate the zone hosting the builder to a different machine. In the process, swap was inadvertently reduced from 32 GB to 4 GB. With WORKDIR residing in /tmp (tmpfs), VM shortage could lead to those errors. I've now restored the previous swap size, which should make the problem vanish, like it did for the last year.