…art text

Now events and phases of the build are represented by a span, with a
stop and start, and each has timing and a success status.

These will visualize nicer (later) and will be able to be logged and
queried in Cloud Datastore + BigQuery.

Also, remove the overloaded and not-to-be-used "rev" parameter from
the GetBuildlet interface. An upcoming change disconnects the
buildlet's lifetime from that of a single build, so the rev isn't even
known at the time of the GetBuildlet call; the buildlet will be handed
out to the most worthy current waiter (and the rev might not have even
existed at the tim the GetBuildlet call started)

Updates golang/go#12669

Change-Id: I02b28fea8bbbc16dc95c1ea0507418be01aa8ca8
Reviewed-on: https://go-review.googlesource.com/21570
Reviewed-by: Andrew Gerrand <adg@golang.org>

Next step is logging it to datastore/bigquery. Then stats and
visualizing with trace viewer later.

Updates golang/go#12669

Change-Id: I039097791993dbe3d4aceb18d2a08db8299f3c05
Reviewed-on: https://go-review.googlesource.com/21640
Reviewed-by: Andrew Gerrand <adg@golang.org>

These IDs will become the cloud datastore keys for keeping history and logs.

Many more CLs will follow: changing farmer URLs, logging to datastore,
fetching old logs from datastore, etc.

Updates golang/go#13076 (make all logs URLs permanent)
Updates golang/go#12669 (collect logs, stats)

Change-Id: I5b9fd21bf23581c59724b0ed32c8459baa9683f7
Reviewed-on: https://go-review.googlesource.com/21968
Reviewed-by: Andrew Gerrand <adg@golang.org>

For now, just log the process's start time and alive/heartbeat time.

Updates golang/go#12669

Change-Id: I4dd28f7ba761b5487f86a4cdb0f721b2aeb4fa57
Reviewed-on: https://go-review.googlesource.com/22701
Reviewed-by: Andrew Gerrand <adg@golang.org>

Updates golang/go#12669

Change-Id: I62a245492081519ba02776366d633e31d74a3c76
Reviewed-on: https://go-review.googlesource.com/22705
Reviewed-by: Andrew Gerrand <adg@golang.org>

Currently this tool just syncs build data from Datastore to Bigquery.

It'll do more later.

Updates golang/go#12669

Change-Id: Ic544cf112ea8699c86943c32f8887d31c81daf79
Reviewed-on: https://go-review.googlesource.com/29112
Reviewed-by: Quentin Smith <quentin@golang.org>

Updates golang/go#12669

Change-Id: Ic7dcc9876557abdb13cd2e6024eb98cdff97c9bb
Reviewed-on: https://go-review.googlesource.com/30631
Reviewed-by: Ian Lance Taylor <iant@golang.org>

We've been logging event spans to datastore for years, but I'd lost
this CL and just found it back. This does two things: syncs the
datastore logs to BigQuery, and starts to use the from-BigQuery timing
info in the coordinator for scheduling sharded tests.

The plan was to have a job occasionally do a BigQuery query and write
out the results to a CSV file on GCS. The code to read that CSV file
is in this CL, but that code path is disabled, so this CL should be a
no-op.

A future change will periodically do the query and write the CSV file,
and then we can start using the new code path and remove the static
map of expected test durations.

Updates golang/go#12669

Change-Id: Ibe5b41d6a3009c2ade8ab728fa1cad646788e621
Reviewed-on: https://go-review.googlesource.com/30716
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

This CL is almost entirely code movement. There is new code in
cmd/coordinator/gce.go.

This library-itizes the buildstats code and makes it used by both
cmd/buildstats (when run by hand) and cmd/coordinator, which now runs
it in a loop in the background.

Running cmd/buildstats by hand is still pretty useless because it
doesn't yet(?) have any query subcommands. It only has the original
-sync command, which is now automatic in the coordinator. But it could
still be useful for debugging or running manually in the staging
environment.

This will permit running BigQuery queries at any time, without Brad or
somebody first manually running cmd/buildstats -sync.

This also means we'll be able to get rid of a bunch of the static (and
stale) timing information in the coordinator code and use empirical
numbers instead for buildlet creation times and per-test durations.
This will help the test sharding schedule things better and result in
faster builds.

Updates golang/go#12669

Change-Id: If9a3b414f8d458a65305c4916ea16d59e5a3971c
Reviewed-on: https://go-review.googlesource.com/53072
Reviewed-by: Sarah Adams <shadams@google.com>