x/build: builds waiting on resources shouldn't be in building state #10716
Labels
Builders
x/build issues (builders, bots, dashboards)
Milestone
Comments
|
A related problem of the current design is that
the next commit that gets the machine is non-deterministic.
We should apply weights to the waiting builds so
that the most recent one gets the machine first.
This problem is esp. apparent on the slowest iOS builders.
|
|
CL https://golang.org/cl/10921 mentions this issue. |
bradfitz
added a commit
to golang/build
that referenced
this issue
Jun 11, 2015
There were too many blue gophers too early on build.golang.org. Wait until we have quota first (GCE quota or reverse buildlet pool aquired). Updates golang/go#10716 Change-Id: I076e89f319d68d55a61f3f8faf0242342c5ae825 Reviewed-on: https://go-review.googlesource.com/10921 Reviewed-by: Andrew Gerrand <adg@golang.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
/cc @crawshaw @adg
The reverse BuildletPool impl is cheating a bit or interacting weirdly with the rest of the coordinator: instead of blocking on returning a buildlet, it instead "starts" a bunch of builds even if there's not an idle iPhone available. And then each builds looks like it's in progress on build.golang.org (with happy blue gopher), but in reality they're all just stuck, for 4 hours (!?) here:
Arguably, the GCE pool impl might do this too, but it happens much less often so it's been easier to miss and ignore.
We should have make quota a more first-class concept within the coordinator and have different states for "waiting" vs "building". And when quota becomes available, we should prioritize which waiting build gets to go next. Right now the one who gets lucky and grabs a mutex gets to go next.
Then, once we have two states, we should make build.golang.org have two colors of gophers: gray for waiting, and blue for actively building.
The text was updated successfully, but these errors were encountered: