@mundaym

I see 23 entries of linux-s390x-ibm under active builds. They're all started anywhere between 51 and 46 hours ago, and stuck on this step:

waiting_machine_in_use 

That might (or might not) be related.

The buildbot is still running normally (as far as I can tell) but no builds are happening. Suggestions?

@billotosyr @mundaym Can you try restarting it (the buildlet process)? Perhaps it's running but disconnected, and hasn't detected that.

You can also try updating to the latest version (i.e., go get -u golang.org/x/build/cmd/buildlet).

restarted buildlet process - doesn't seem to have had any effect.

I'm pretty sure its not a version thing -- as the buildlet was working up untl a benign unrelated CL yesterday ("skip TestGcSys on Windows").
Our machine had run out of space, we fixed it, and it started building again, but somehow the queue is too big perhaps? It just seems clogged. Any thoughts on getting it unclogged?

Our machine had run out of space, we fixed it, and it started building again, but somehow the queue is too big perhaps? It just seems clogged. Any thoughts on getting it unclogged?

Great to hear that it's running again!

Do you know if the machine ran out of space due to the Go builder work, or was it for unrelated reasons?

About the queue, after talking to @bradfitz about it, I don't think there's an existing way to clear it out. Sorry about that. We have an open bug for a scheduler (#19178) and that would be helpful here (e.g., prioritize more recent commits). Let's keep an eye on how the builder runs, and I'll investigate what else can be done to help.

Specifically, it's build cache that is filling up the drive. My current workaround is to use
'go clean -cache '
as root, frequently. Perhaps I will have to turn off the build cache.

I thought the buildlet wiped its work directory environment per run?

Our buildlet doesn't do that. Maybe because it's an older version.

No, I was just misremembering. But if that helps your environment & wrapper process that's managing keeping it alive, we could add it to the buildlet.

Well, I think it would help us, yes. The build cache is consuming about 2GB per day, and the drive it's on is small enough that I am now cleaning it daily. (This is a new thing, as I never had to clean it before.)

Change https://golang.org/cl/144637 mentions this issue: cmd/buildlet: set up & clean TMPDIR and GOCACHE for child processes

Okay, this the new s390x buildlet binary is pushed.

I'll watch it for any new problems.

It'll keep itself cleaned for new stuff, but you might have to clean legacy messes.

Thank you! There may be an issue though:
https://build.golang.org/log/4972bb50dbc4c99c530bc3c9bea9edffb70d3430

Hmm, weird.

I'll debug in a few hours.

Change https://golang.org/cl/144858 mentions this issue: cmd/buildlet: restore tmp/gocache dirs after recursive delete of world