This is the first in a series of patches which include an
effort to refactor how the cmd/coordinator is structured. In order
to move buildlet pools into their own package, it would facilitate
the move to have spanlog in it's own internal package. This CL will
be followed by one which removes cmd/coordinator/spanlog and
changes the path of the package for all the callers.

Updates golang/go#36841

Change-Id: Ic310a38f807a3099815e0bdb988edc42b2ca5c85
Reviewed-on: https://go-review.googlesource.com/c/build/+/226843
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

This removes the spanlog package located inside the
coordinator package. The spanlog package has already been
copied over to the internal package. All imports in the build
repository have been changes to point to the internal spanlog
package.

Updates golang/go#36841

Change-Id: I84499bcfe782c566b12dabcbfa743cd8ce559f18
Reviewed-on: https://go-review.googlesource.com/c/build/+/227023
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

This CL creates the internal/coordinator/pool package intended to
contain all buildlet pool implementations. In order to keep this
change small and carefully discover where the interactions are
between the gce buildlet pool and the rest of the coordinator
are, this change only moves the gce buildlet over to the new
package.

The next steps will be to move the rest of the buildlet pools
over to this package. After that we will restructure the
implementations themselves in order to increase test coverage
and increase the ease of testing.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: If82ae1b584bd77c697aa84fadf9011c9e79fa409
Reviewed-on: https://go-review.googlesource.com/c/build/+/227141
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

This is a set in a series of steps which will move everything buildlet
pool related into a pool package.

Updates golang/go#36841

Change-Id: I8efb1f94c7b929be559004d9f455bca0370c7800
Reviewed-on: https://go-review.googlesource.com/c/build/+/227768
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

This is a set in a series of steps which will move everything buildlet
pool related into a pool package.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: Ic7a0ccd7838345036df2e72b13084070541cb63c
Reviewed-on: https://go-review.googlesource.com/c/build/+/227769
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

This change moves all package level accessors into a struct as
suggested by Alex:
https://go-review.googlesource.com/c/build/+/227141/6/internal/coordinator/pool/gce.go#227
This will make it easier to move away from global state in
future changes.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: I005884ccd206fe49651d31ef1d3d336fac9c3d5f
Reviewed-on: https://go-review.googlesource.com/c/build/+/227920
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

This adds the ability for linux-arm64 builder images to be
created on AWS. Instead of writing a bash script which would create
an image with all of the dependencies on it, this experiments with
using packer to create the image.

The image is configured to install and daemonize rundockerbuildlet.
Rundockerbuildlet will download and run the latest stage0 image.
Stage0 then downloads the latest buildlet and begins processing jobs.

A follow-up change will enable rundockerbuildlet to initialize the
buildlet without setting it in reverse mode.

Updates golang/go#36841

Change-Id: Iee07c2600e2b91d34f5e6a1e062f763833d79904
Reviewed-on: https://go-review.googlesource.com/c/build/+/228799
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

This change adds an AWS buildlet client which allows us to
create EC2 instances on AWS. With this change we have also
moved a portion of the gce creation logic into a helper
function which allows multiple clients to use it. Metadata
for the instances are stored in the user data fields.

The creation of a buildlet pool and modifications to
rundocker buildlet be made in order to enable this change.

Updates golang/go#36841

Change-Id: Ice03e1520513d51a02b9d66542e00012453bf0d9
Reviewed-on: https://go-review.googlesource.com/c/build/+/232077
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

The buildlet client creation will either timeout at the current
default timeout period or the timeout set in the context. The
existing effective timeout is the five seconds set in the
probeBuildlet function.

Fixes golang/go#38956
Updates golang/go#36841

Change-Id: I09e96e2c5abcc45ccd535596104c52998ddb0d7a
Reviewed-on: https://go-review.googlesource.com/c/build/+/232997
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

FreeBSD 9.3, FreeBSD 10.x, and OpenBSD 6.0 have
reached end-of-life and are no longer supported.
Go 1.12 was the last release¹ that ran on FreeBSD 10.x.
Go 1.10 was the last release² that ran on OpenBSD 6.0.

These builders have long since been disabled and won't be needed in
the future, so remove them. This work helped detect golang/go#40563.

Modify TestHostConfigsAllUsed to report errors instead of warnings
when it finds unused host configs. Make a map of known exceptions
with rationale or a linked issue for each entry.

¹ https://golang.org/doc/go1.12#ports
² https://golang.org/doc/go1.10#ports

Updates golang/go#36841.
Updates golang/go#19938.

Change-Id: Id7fcdc671bfacf7e70174f4168c9c9cc46ba53d7
Reviewed-on: https://go-review.googlesource.com/c/build/+/246639
Reviewed-by: Alexander Rakoczy <alex@golang.org>

This change modifies the WaitUntilVMExists method and sets it to
not be exported. It will never be directly called by anything outside
of the EC2Client.

Updates golang/go#36841

Change-Id: If6dfea8c4c191036c8fadd9ad8767e85e55cd224
Reviewed-on: https://go-review.googlesource.com/c/build/+/247400
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

This change stops explicitly selecting a random availability zone upon
EC2 vm creation if one is not set. The EC2 api will select an
availability zone with capacity for the request when an availability
zone is not set. The caller maintains the ability to set an
availability zone if needed.

Updates golang/go#36841

Change-Id: I87d4e8e66fcfadfedbe584a2c016696dc4d62062
Reviewed-on: https://go-review.googlesource.com/c/build/+/247901
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

This change adds a ledger to the coordinator pool package. The ledger
is used to store a record of EC2 instance types created and their
attributes. It also keeps a record of resource usage for EC2 instances
in order to ensure that we do not exceed the resource limits for the account.

Updates golang/go#36841

Change-Id: I6a0afdb31c8e3a83634e7c1fc8b2b733b7a50c01
Reviewed-on: https://go-review.googlesource.com/c/build/+/247906
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

The EC2 buildlet pool added by this commit will manage the lifecycle
of buildlets running on EC2. EC2 VMs will only be created for the
ARM64 architecture. As VMs are requested, the pool will ensure that
there are enough resources for the VM to be created and keep track of
the VMs created. Once a VM is destroyed, the resources will be made
available for other VMs. This pool will only keep instances as they
are needed.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: Ic777485c0b0a69ec13726c58b49e9fdc1df4777e
Reviewed-on: https://go-review.googlesource.com/c/build/+/247907
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

Before destroying an untracked EC2 instance, verify that it is a
buildlet. Non buildlets will be destroyed manually as they are most
likely instances created while an operator is debugging a problem.

For golang/go#36841

Change-Id: Id0f65192ce4e72f9ecc495dd3e94750724091b51
Reviewed-on: https://go-review.googlesource.com/c/build/+/249121
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

Add the host-linux-arm64-aws builder. The HostConfig IsVM method has
been modified to support the concept that an EC2 VM can be created
which runs the buildlet binary in a container. This configuration
would have both a VM image and a container image set in the host
configuration.

For golang/go#36841

Change-Id: I092b48b1dda24d24bcd0b10dff0b2126f5f43017
Reviewed-on: https://go-review.googlesource.com/c/build/+/249118
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

When rundockerbuildlet is running on an EC2 instance, the
configuration should be pulled in from the metadata. This is part of
the effort to move some configuration for builders over to the
coordinator.

For golang/go#36841

Change-Id: Icbf0e73301faadd06c87f02aa1172bd0e6f87624
Reviewed-on: https://go-review.googlesource.com/c/build/+/252317
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

There is no need to set availability zones for EC2 resources at this
time. The resources currently in use are available to all zones within
a single region.

For golang/go#36841

Change-Id: Ifcb2c3404fba2489741c31fe63ad4e7747639ad9
Reviewed-on: https://go-review.googlesource.com/c/build/+/255359
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>

This adds a polling function that is tested and can be reused by
callers in the x/build repository. This was suggested in the review of
CL 247907. The polling function replaces the pollers used in the EC2 buildlet pool.

For golang/go#36841

Change-Id: I120ceb83e2740f0bdc5ee2423e0edd3ad727bf4b
Reviewed-on: https://go-review.googlesource.com/c/build/+/255358
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Alexander Rakoczy <alex@golang.org>

This change updates which source ami is used as a base for the
creation of the linux-arm64-aws builder. Packer will now search for
the latest version of the arm64 Debian image to use as the source ami.

For golang/go#36841

Change-Id: I036dbaf7ee65d4e7028cd6a0cdf6710bf898fdd2
Reviewed-on: https://go-review.googlesource.com/c/build/+/266697
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>