Currently all builds start and think they're running, but most are
just fighting over a mutex to grab a builder. That will be fixed, but
in the meantime it's nice to see what's actually working vs what's
waiting on e.g. arm5 hardware which won't be available for hours.

This is a baby step towards more monitoring. Currently this is just HTML
output, but the same data could be exported via JSON or something else later
for graphing.

Updates golang/go#19178 (add a buildlet scheduler)
Updates golang/go#15760 (monitor everything)

Change-Id: I36e16ea0919afe8023fe7fedd981f2e857f0d6df
Reviewed-on: https://go-review.googlesource.com/40397
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

Benchmarks are treated as unit tests and distributed to the test
helpers, which allows them to fit in our 5m trybot budget.

Currently we only run the go1 and x/benchmarks. Running package
benchmarks is a TODO.

This feature is disabled by default, and is enabled by the
"farmer-run-bench" project attribute.

Updates golang/go#19178
Updates golang/go#19871

Change-Id: I9c3a14da60c3662e7e2cb4e71953060915cc4364
Reviewed-on: https://go-review.googlesource.com/38306
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

…lets

The problem is especially severe on the slow builders,
like darwin/arm64, which can sometimes not have run
any commit from the past 12 hours and still pick an old
commit for its next trial. It would be far better for it to
prefer new work.

It's possible that this new logic should be disabled for
some of the auto-scaling builders, but for now it seems
like we can try it for all of them and see if that's OK.

Update golang/go#19178.

Change-Id: I32cc67c0c2c84130b40b250675b40aadb4a0a681
Reviewed-on: https://go-review.googlesource.com/70430
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Sarah Adams <shadams@google.com>

Updates golang/go#19178

Change-Id: I24aa368df01a85259b53d6cfb08de7ab3a80e4fe
Reviewed-on: https://go-review.googlesource.com/132076
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

Optimizations and tuning remain, but this should be tons better than
what we had before (random).

Updates golang/go#19178

Change-Id: Idb483a4c4209a012814322cc8b37b966ee4681de
Reviewed-on: https://go-review.googlesource.com/c/build/+/205078
Reviewed-by: Bryan C. Mills <bcmills@google.com>

Optimizations and tuning remain, but this should be tons better than
what we had before (random).

Updates golang/go#19178

Change-Id: Idb483a4c4209a012814322cc8b37b966ee4681de
Reviewed-on: https://go-review.googlesource.com/c/build/+/205078
Reviewed-by: Bryan C. Mills <bcmills@google.com>

It now passes thousands of iterations in race mode.

Updates golang/go#19178

Change-Id: I210277abd084bdfd3c7ada538189722ff9543e3f
Reviewed-on: https://go-review.googlesource.com/c/build/+/207079
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>

The HTML status for the coordinator was way too long. For pending
builds, only show a single line, and render their state as
"waiting_for_machine" rather than "running". And for active builds,
only show the last few lines of status on the home page. People can
click for details.

Then add a scheduler status section too.

I'm also stashing away a build's SchedItem for now (with a little
refactoring to break up a long method), so a future CL can tell people
where a build is in line to get a buildlet.

Updates golang/go#19178

Change-Id: I2f37982ea3c7ee4a6581464117ae533499eba6a4
Reviewed-on: https://go-review.googlesource.com/c/build/+/207179
Reviewed-by: Bryan C. Mills <bcmills@google.com>

This is in its own CL for now so it's easy to revert.

Updates golang/go#19178

Change-Id: I2eb66e3c8a6e75a8039077401231577c8e7bfdc8
Reviewed-on: https://go-review.googlesource.com/c/build/+/207180
Reviewed-by: Bryan C. Mills <bcmills@google.com>

…code, TODOs

Updates golang/go#19178

Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493
Reviewed-on: https://go-review.googlesource.com/c/build/+/207178
Reviewed-by: Bryan C. Mills <bcmills@google.com>

…vant

Updates golang/go#19178

Change-Id: Ib59eabafc589ce66948c91c7f15cbccae2d2733d
Reviewed-on: https://go-review.googlesource.com/c/build/+/207418
Reviewed-by: Bryan C. Mills <bcmills@google.com>

…tart

Updates golang/go#19178

Change-Id: I054c852db398a9474e53254c42eb5f77c44fe348
Reviewed-on: https://go-review.googlesource.com/c/build/+/207464
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

… scheduler

The branch is not yet used in this CL, but the scheduler has it now
and can use it easily in the future.

Updates golang/go#19178

Change-Id: I6abab826a8668cb091d0face8184f28d08421722
Reviewed-on: https://go-review.googlesource.com/c/build/+/208277
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>