On 18 Dec 2019, at 10:54, Bryan C. Mills ***@***.***> wrote: This failure mode is now happening on every build. @esnolte <https://github.com/esnolte>, you are listed as the owner of that builder here <https://github.com/golang/build/blob/6b2867eab46056b7c8e75fbfadfa40dc43dbdf91/dashboard/builders.go#L217>. Could you please investigate? I wonder if the problem is somehow that multiple builds are running at the same time. (A similar failure mode in #35749 <#35749> turned out to be an older builder running concurrently with the current one.)

On 18 Dec 2019, at 12:11, Erik Nolte ***@***.***> wrote: The builder is in a loop filling the logs and eventually the filesystem with: Dec 18 19:01:57 localhost logger: stage0: 2019/12/18 19:01:57 downloading https://storage.googleapis.com/go-builder-data/buildlet.linux-arm-arm5 <https://storage.googleapis.com/go-builder-data/buildlet.linux-arm-arm5> to ./buildlet.exe ... Dec 18 19:01:57 localhost logger: stage0: 2019/12/18 19:01:57 downloaded ./buildlet.exe (12317133 bytes) Dec 18 19:01:57 localhost logger: stage0: 2019/12/18 19:01:57 downloaded buildlet in 200ms Dec 18 19:01:57 localhost logger: 2019/12/18 19:01:57 buildlet starting. Dec 18 19:01:57 localhost logger: 2019/12/18 19:01:57 unlinkat ***@***.***/glass_test.go: ***@***.***/glass_test.go:> permission denied Here is the service we're running: ***@***.***:/home/builder# cat /etc/service/stage0/run #!/bin/bash set -e SCRATCH=~builder/stage0scratch export TMPDIR=${SCRATCH}/tmp export WORKDIR=${SCRATCH}/workdir mkdir -p ${TMPDIR} mkdir -p ${WORKDIR} chown -R builder:builder ${SCRATCH} cd ${SCRATCH} export GO_BUILDER_ENV=linux-arm-arm5spacemonkey export GO_TEST_TIMEOUT_SCALE=5 export USER=builder export HOME=/home/builder exec bash -c "setuidgid builder /usr/local/bin/stage0 2>&1 | logger"k The builder is a binary installed in 2017: ***@***.***:/usr/local/bin# ls -l total 4912 -rwxr-xr-x 1 root root 5013776 Aug 29 2017 stage0 ***@***.***:/usr/local/bin# file stage0 stage0: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped ***@***.***:/usr/local/bin# sha1sum stage0 74ae93f72bcb193778383fd7113a5310f74c2e7c stage0 Should I delete stage0scratch and see if that helps or am I running an old builder and need to upgrade it? - Erik > On 18 Dec 2019, at 10:54, Bryan C. Mills ***@***.*** ***@***.***>> wrote: > > This failure mode is now happening on every build. > > @esnolte <https://github.com/esnolte>, you are listed as the owner of that builder here <https://github.com/golang/build/blob/6b2867eab46056b7c8e75fbfadfa40dc43dbdf91/dashboard/builders.go#L217>. Could you please investigate? > > I wonder if the problem is somehow that multiple builds are running at the same time. (A similar failure mode in #35749 <#35749> turned out to be an older builder running concurrently with the current one.) >

On 19 Dec 2019, at 07:41, Bryan C. Mills ***@***.***> wrote: That doesn't seem to have fixed the problem. (https://build.golang.org/log/08d3ac2849287e18678df797c542b7329dab3d3e <https://build.golang.org/log/08d3ac2849287e18678df797c542b7329dab3d3e>) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35740?email_source=notifications&email_token=ADY27VMKXKRTK2HR6ZV35DLQZOB2RA5CNFSM4JQDMVCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHJZ3AA#issuecomment-567516544>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADY27VMKJ5CK5SQ6OGPDG4TQZOB2RANCNFSM4JQDMVCA>.

On 19 Dec 2019, at 11:42, Dmitri Shuralyov ***@***.***> wrote: I've unplugged a rogue builder we no longer have access to (go-builder-3). It looks like the other 2 builders are passing tests. By now, this seems to be accurately reflected under the "Reverse pool by host type (in use / total)" section of https://farmer.golang.org/#pools <https://farmer.golang.org/#pools>: host-linux-arm5spacemonkey: 2/2 (1 missing) I'm not sure why it was showing something else before. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35740?email_source=notifications&email_token=ADY27VJ6QHJGU7A5EBVH7P3QZO6C3A5CNFSM4JQDMVCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHKRGIA#issuecomment-567612192>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADY27VKLN4B6LLK4MIMUF4TQZO6C3ANCNFSM4JQDMVCA>.

On 27 Dec 2019, at 14:45, Brad Fitzpatrick ***@***.***> wrote: IIRC, the arm5 builders start with a cross-compiled make.bash from a linux-amd64 machine. Perhaps the two environments have drifted? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.