Running benchmarks has been disabled since 2018. Remove all the code to
keep things more maintainable and understandable.

We will be adding new benchmarking support soon, and may reuse some of
this code, but don't want half-working code adding confusion.

For golang/go#49207.

Change-Id: I11d52b0315bed4d91651c162af11853895012868
Reviewed-on: https://go-review.googlesource.com/c/build/+/354315
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

The coordinator is getting support for running the benchmarks in this
repository. Since the benchmarks and interface are in flux, encoding all
of the details of running Go tests, bent arguments, etc into the
coordinator will likely cause churn and frustrating migration issues.

Instead, add cmd/bench which serves as the simple entrypoint for the
coordinator. The coordinator runs cmd/bench with the GOROOT to test
(eventually multiple GOROOTs), and this binary takes care of the
remaining details.

Right now, we just do a basic go test golang.org/x/benchmarks/... and
simple invocation of bent. Note that bent does not pass without
https://golang.org/cl/354634.

For golang/go#49207

Change-Id: I5c9cf89540cab605c0a64e17af85311d37985c25
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/359854
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>

Add initial support for running the performance tests from x/benchmarks.
Since there are a variety of different test suites (some `go test`,
bent, etc), x/benchmarks provides a basic wrapper command,
golang.org/x/benchmarks/cmd/bench which know the minute details. The
coordinator just needs to run that one command.

This build mode is limited to builds of x/benchmarks on builders with
RunBench set to true. Currently there are none, a future CL will add the
initial such linux-amd64 builder.

For golang/go#49207

Change-Id: Ie006ec4a3757a5c2fed0925a3f9eb91edeaa5224
Reviewed-on: https://go-review.googlesource.com/c/build/+/361417
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

The performance test output contains benchfmt-formatted benchmark
results. Upload the output wholesale to perfdata.golang.org for
long-term storage and analysis.

The results for now are a bit rough, as the output may also contain
unrelated output that lines that look like benchfmt. For example. "go:
downloading github.com/BurntSushi/toml v0.3.1" adds a "go" label with
the value "downloading ...". In the future, we will ideally filter these
a bit better (perhaps in x/benchmarks/cmd/bench).

For golang/go#49207

Change-Id: Ifd2512c93902a74f9040db0f9d0c600348fc1849
Reviewed-on: https://go-review.googlesource.com/c/build/+/361418
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>

Add a new builder to run the x/benchmarks performance tests on
linux-amd64.

For now, this runs on a GCE C2 instance type, as these instances have
well-defined, consistent CPUs and other server architecture components.

In basic noise testing, even standard VMs of this type appear to be
fairly low noise. As we gain experience with actual monitoring, we may
change this to a sole-tenant VM type or even a dedicated machine if
necessary.

For golang/go#49207

Change-Id: I17eaeeb5349af925249940bebd5b860a2579e6df
Reviewed-on: https://go-review.googlesource.com/c/build/+/354311
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

We currently use E2, C2, and N2 instances on GCE. C2 and N2 instances
have their own quotas, which are accounted separately from the CPUS
quotas.

This could probably be cleaned up to keep track of all CPU quotas and
handle more instance types, but this should work for the time being.

See: https://cloud.google.com/compute/quotas#cpu_quota

For golang/go#49207

Change-Id: Ida1e8de3c857560637095d57e972bca7222284ed
Reviewed-on: https://go-review.googlesource.com/c/build/+/361656
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>

Since this builder doesn't build the go repo, it will be waiting forever
for a snapshot. Instead, just build Go for each run.

For golang/go#49207

Change-Id: I34a73b507278db402c478b4f5956633996772aae
Reviewed-on: https://go-review.googlesource.com/c/build/+/361754
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

Make rsync optional with fallback to cp.

Remove use of /usr/bin/time and replace with measuring time directly
from Go.

For golang/go#49207

Change-Id: Ief5a7a90f9460ddec1d5a51c99d4a534e38a5d9c
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/361734
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: David Chase <drchase@google.com>

These make it possible to tell what was run, as well as a convenience
field stating whether this was a post-submit build or a trybot run.

For golang/go#49207

Change-Id: Iba979bcfd5a3bbdc11e2df0b8de4094cc7212356
Reviewed-on: https://go-review.googlesource.com/c/build/+/361874
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

If BENCH_BASELINE_GOROOT is set, additionally benchmark that toolchain.
The benchfmt label 'toolchain' differentiates the 'experiment' and
'baseline' toolchains.

For golang/go#49207.

Change-Id: I737fa56786dc482172942462c5776c4c2773c0c5
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/376096
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>

When benchmarking, we want to benchmark both the toolchain under test
(i.e., buildStatus.Rev) as well as an older "baseline" toolchain, which
will be compared against.

For now, the baseline toolchain is the latest stable release. In the
future we may want to update more frequently, but this is a simple
starting point.

This CL determines the baseline toolchain commit for a given test and
installs it on the buildlet at BENCH_BASELINE_GOROOT.
golang.org/x/benchmarks/cmd/bench is responsible for utilizing the
baseline toolchain. CL 376096 is the corresponding change to cmd/bench.

Most of the baseline toolchain logic is limited to runBenchmarkTests().
In theory, it logically fits a bit better with the rest of the toolchain
logic in build() et al, but keeping it limited to runBenchmarkTests()
helps keep the common build() path from getting much more complex for a
minor edge-case feature.

For golang/go#49207.
For golang/go#48803.

Change-Id: Id63f8333cf9d1ff952850c3347e999b5e98f7294
Reviewed-on: https://go-review.googlesource.com/c/build/+/376634
Reviewed-by: Alex Rakoczy <alex@golang.org>
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

The coordinator runs tests on a freshly booted VM which may still be
running background boot tasks when bench starts.

For minimal noise, wait for the system load average to drop (indicating
background tasks have completed) before continuing with benchmarking.

For golang/go#49207.

Change-Id: I8df01592fea31d49eae54074213e202b21d5728a
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/362375
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

Allow individual host configurations to override the VM delete timeout
if they are using for longer than normal builds.

For golang/go#49207.

Change-Id: I9c5c80e5ee7dac2375cff17c64871ae2211f6309
Reviewed-on: https://go-review.googlesource.com/c/build/+/378334
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>

Right now we "iterate" over toolchains (GOROOTs) at the outer-most part
of the tool, but bringing that in closer lets us do things like only
build the benchmarking tools once.

This change also introduces abstractions around the Go tool from the
Sweet tool to simplify and deduplicate some code. For instance, building
bent currently fails with the baseline GOROOT because the GOROOT
environment variable isn't set correctly, but the "gotest" benchmarks
do.

For golang/go#49207.

Change-Id: I6816e1112174f951d3bc22c2b1033b8e98dc0327
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/378214
Reviewed-by: Michael Pratt <mpratt@google.com>
Trust: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>

The benchmarks in the perf builder may take several hours to complete.
Extend the VM deletion timeout so that they stick around long enough to
complete benchmarking.

For golang/go#49207.

Change-Id: I3e9d2a1df657406ef0f80b9c0cb713df3b716ca8
Reviewed-on: https://go-review.googlesource.com/c/build/+/378336
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>

Some benchmarks in x/benchmarks from external sources wrap the go tool
in make. Add make to the linux/amd64 builders where these benchmarks will
run.

For golang/go#49207.

Change-Id: I4ea16c0aa63d1b520c61d0a2b9dabffdd8bb7094
Reviewed-on: https://go-review.googlesource.com/c/build/+/382894
Trust: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>

For golang/go#49207.

Change-Id: Ib18c5f574e30333a7d9d80019e26d6a565f4db1e
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/378274
Reviewed-by: Michael Pratt <mpratt@google.com>
Trust: Michael Knyszek <mknyszek@google.com>

For golang/go#49207.

Change-Id: I83fa87a603cf26ed61d324975166388db1801487
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/382097
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Trust: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

This is consistent with the format used by perfdata.golang.org for
upload-time and x/perf/cmd/bench for runstamp.

For golang/go#49207.

Change-Id: I0c800629c23eb830803d3017806ca6c9c8907b87
Reviewed-on: https://go-review.googlesource.com/c/build/+/397655
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>