Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: linux-arm-arm5spacemonkey build stalled: no space left on device #21351

Closed
s-mang opened this issue Aug 8, 2017 · 12 comments
Closed
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Milestone

Comments

@s-mang
Copy link
Contributor

s-mang commented Aug 8, 2017

ERRORs in stackdriver:
2017/08/08 19:36:18 {linux-arm-arm5spacemonkey e0e7c03 } failed: failed to put snapshot to buildlet: 500 Internal Server Error; body: error writing to /home/builder/stage0scratch/tmp/workdir/go/test/fixedbugs/bug338.go: write /home/builder/stage0scratch/tmp/workdir/go/test/fixedbugs/bug338.go: no space left on device

Logs: https://farmer.golang.org/temporarylogs?name=linux-arm-arm5spacemonkey&rev=c4e29bbd3813be43b3905e17a2796cea23dd66d4&st=0xc44a7c8600

go-builder-3 (10.240.0.18:56501) version 11, host-linux-arm5spacemonkey: connected 2h57m50.4s, working for 2h57m50.4s

cc @bradfitz @jessfraz @cybrcodr @zombiezen @shantuo

@gopherbot gopherbot added this to the Unreleased milestone Aug 8, 2017
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Aug 8, 2017
@jessfraz
Copy link
Contributor

jessfraz commented Aug 8, 2017

bigger disks or more cleanup of disk space?

@bradfitz
Copy link
Contributor

bradfitz commented Aug 8, 2017

None of the cced people own these machines. See the farmer builders page for the owner.

@s-mang
Copy link
Contributor Author

s-mang commented Aug 8, 2017

@zeebo

@bradfitz
Copy link
Contributor

bradfitz commented Aug 8, 2017

@zeebo, let us know what you find. The buildlet should be cleaning up after itself after (or rather, before) each build.

But it's possible you're not setting the -workdir flag, which means it's picking a new temp dir out of /tmp for each build, and then if your builds fail midway though (or rather, we restart the coordinator mid-build and your buildlet process restarts) then it might be picking a new workdir and keeping the old one around if your system /tmp dir doesn't clean or clean fast enough.

I think the best fix is for your builder looper to pass an explicit -workdir flag.

@zeebo
Copy link
Contributor

zeebo commented Aug 9, 2017

We set TMPDIR to something, and yep it's full of things.

I'll change the scripts to pass -workdir and clean it up.

@zeebo
Copy link
Contributor

zeebo commented Aug 9, 2017

Ok I've looked in to this some. I don't see a way that I can from my environment pass -workdir to the buildet command. the stage0 command handles running the buildet for me, and I see no way in either stage0 or buildet pass it the flag.

Is the next step to modify one of these two binaries to allow me to pass it in through some environment variable or something? Is my setup wacky?

I've temporarily cleaned out the directories.

@s-mang
Copy link
Contributor Author

s-mang commented Aug 18, 2017

@zeebo sorry for my delay.
yes, please submit a CL to modify cmd/buildlet/stage0/stage0.go.
there is a switch towards the bottom of main() (L167) which would be appropriate to add to.

LMK if you have questions. please add me as the reviewer when your cl is up.

@gopherbot
Copy link

Change https://golang.org/cl/58130 mentions this issue: cmd/buildet/stage0: allow linux-arm-arm5spacemonkey to pass workdir

@zeebo
Copy link
Contributor

zeebo commented Aug 28, 2017

I'm going to attempt to move the conversation here vs the above CL to keep the conversation together. After updating the stage0 binary, it seems to me that other changes to the infrastructure have caused problems, requiring a rebuild of the hosted buildlet binary. After that, the build key seems to be invalid.

bad key for mode "host-linux-arm5spacemonkey"

In case it helps figure out the issue sooner, I had to rename the /.gobuildkey to /.gobuildkey-host-linux-arm5spacemonkey to get that far.

I assume I just need to get a new key sent to me? I don't know how public these keys are, but just in case since it's called a "key", I'm available at the email in my github profile or as @zeebo on the gophers slack.

@s-mang
Copy link
Contributor Author

s-mang commented Aug 29, 2017

Hey @zeebo sending you a new key via email now.

@zeebo
Copy link
Contributor

zeebo commented Aug 29, 2017

Alright, they're all running with the new key, and the flag is being passed.

Thanks!

@s-mang
Copy link
Contributor Author

s-mang commented Aug 29, 2017

excellent. np.

@golang golang locked and limited conversation to collaborators Aug 29, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Projects
None yet
Development

No branches or pull requests

5 participants