This change adds the signing server used to request artifact signing.
An interface has been added to the sign package in order to create
different implementations of the signing service if needed. With the
GRPC signing service, the client creates a bidirectional connection
with the server. The server then sends well specified signing requests
to the client.

Updates golang/go#54303
Updates golang/go#53632

Change-Id: I062f7db716edba810b6c1dab122b2a10c1c18923
Reviewed-on: https://go-review.googlesource.com/c/build/+/422598
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

This makes the release server accessible via build.golang.org.

Updates golang/go#53632
Fixes golang/go#54303

Change-Id: I7628ef8f46b52ce27a0b41ce4e85108388cf64e9
Reviewed-on: https://go-review.googlesource.com/c/build/+/424915
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Joedian Reid <joedian@golang.org>

This adds the GRPC release server to the main relui HTTP server.

Updates golang/go#53632
Updates golang/go#54303

Change-Id: I244ac02db1c2647e3f9f0f8f72a89467f07d1881
Reviewed-on: https://go-review.googlesource.com/c/build/+/424914
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>

This change is a no-op other than moving messages and fields around
to make them easier to make sense of and to navigate when reading
from top to bottom. The next CL makes logical changes.

For golang/go#53632.

Change-Id: I03bf5e31f1ee2845b6e6014f610b3e07540b0159
Reviewed-on: https://go-review.googlesource.com/c/build/+/428056
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

Development and testing of the release signing workflow helped us find
ways to adjust the wire protocol to be a better fit for our needs.

The responsibility to come up with a job ID for future requests is moved
to the initial sign request creation transaction, replacing the need for
taskName and retryCount parameters.

Have just one build type for macOS signing because we no longer need to
keep the AMD64 and ARM64 versions separate.

Add a message and endpoint for marking a previously started signing job
as obsolete, so it'll be possible to do this automatically when a relui
release workflow happens to be stopped by a release coordinator.

Update the sign server and its tests accordingly.

For golang/go#53632.

Change-Id: I99081b0a0f8140b68e1a4b20bc5e422516052e28
Reviewed-on: https://go-review.googlesource.com/c/build/+/428057
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>

This change implements functionality to build a macOS installer using
Xcode tools on the remote buildlet. This is somewhat analogous to how
the existing releaselet command builds a Windows installer.

Templates are used to generalize it for arm64 and amd64 architectures.
At this time all supported Go versions share the same minimum macOS
version requirement. When that changes, it'll be possible to use these
templates to set an appropriate value for each one. (In the past,
we always used one value for all installers, making it less precise.)

Otherwise this implements the same installer as previously constructed
during the signing process.

This code isn't used yet. The next CL in the stack starts to use it.

For golang/go#53632.

Change-Id: I53b4ea68cd9cbb41b6bc6cacd136322284e1fae6
Reviewed-on: https://go-review.googlesource.com/c/build/+/443060
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

This new dry-run workflow starts to exercise the new signing service.

Modify the naming pattern for scratch files to place the random number
in front of the filename to avoid modifying the file extension.

For golang/go#53632.

Change-Id: I799e4f9d50b22130a91428477c675c562952f441
Reviewed-on: https://go-review.googlesource.com/c/build/+/443095
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>

Use dedicated buffered channels with capacity of 1 to receive responses
to messages, instead of a local variable and a cancel context. This is
simpler and fixes a race.

The signResponse struct had two places where message ID could be read
from, and some code paths read the wrong one. Keep only the message
and error fields, and factor the message ID out of this struct.

The responseCallback did not seem to benefit from being separated
out of SigningServer; combine the two to simplify the code.

Rename send to do since it waits for the response before returning.

Make the requests channel unbuffered rather than having a capacity
of 1 to simplify the code. It doesn't benefit from being buffered
since operations block until a response is available. This makes the
first request handled the same way as the second, third, etc.

For golang/go#53632.
Fixes golang/go#56297.

Change-Id: I2ae4f774aabe4c4bdae34694dc1ad46896fc44a3
Reviewed-on: https://go-review.googlesource.com/c/build/+/443676
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

The StatusRunning and StatusFailed protobuf messages have a field named
Description, though previously its content wasn't accessible. Return it
to the caller and log it. This way we can figure out whether to keep it
or possibly remove it entirely if it stays unused.

It was useful during initial development and testing at least.

Also modify ArtifactSigningStatus to return a nil error when status
is NotFound, since it's a valid status that the caller can handle.

For golang/go#53632.

Change-Id: I7f9b038d86f985b71300150d1ccc700e209cdf2b
Reviewed-on: https://go-review.googlesource.com/c/build/+/444255
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

I expect successful communication with a signing client to be on the
order of a few seconds, maybe 10 or 20. Some signing status requests
are stalling for 10 minutes.

In such cases, it's better to give up earlier and try again, so cap
individual status requests inside AwaitCondition to a minute at most,
and prefer to try again if that doesn't work out. This should prevent
the watchdog timer from reaching 10 minutes and stopping the task.

Also improve logging to print more useful timing information, and
fix a case where UpdateSigningStatus was always returning nil error.

For golang/go#53632.

Change-Id: Idb340fb6396b31570acd980adf33a52877d60012
Reviewed-on: https://go-review.googlesource.com/c/build/+/444495
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>

Started and Failed are the two possible responses
to a SignArtifact request, so add the missing one.

For golang/go#53632.

Change-Id: I9d0e4ef21b5f32ba3eac780fd8421ed36af8387e
Reviewed-on: https://go-review.googlesource.com/c/build/+/445595
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

Our stream.Recv returns io.EOF when the other end has done a CloseSend,
at which point the bidirectional stream is unhealthy since it can only
send but not receive further messages. Return only non-nil errors from
the errgroup functions so that both of them end as quickly as possible,
to remove the possibility of a send-only stream being long-lasting.

Also return stream.Send errors directly rather than converting them to
an opaque internal error, since it's simpler and doesn't hide possibly
useful error details from the gRPC server.

For golang/go#53632.

Change-Id: I53bfe51161c93f00d088f4a7aae706d55295f026
Reviewed-on: https://go-review.googlesource.com/c/build/+/445855
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

The new tasks to create a macOS installer uncovered a few places where
the fake buildlet was skipping the step of creating directories like
the real one. Fix that so it behaves more like the real one.

For golang/go#53632.

Change-Id: I5882d916edba92e21e25282a70efed6bd49ad17b
Reviewed-on: https://go-review.googlesource.com/c/build/+/447296
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

During testing, it's useful to be able to supply fake system commands.
Add a mechanism that allows doing that to FakeBuildlets. This will be
used by the release test to mock macOS installer building.

For golang/go#53632.

Change-Id: I003b357112b1f5349503d928b7b5f2e3411614a2
Reviewed-on: https://go-review.googlesource.com/c/build/+/447297
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

Add useSignService, a const initially set to false, that controls
whether the new sign.Service is used in the main release workflow,
addSingleReleaseWorkflow, and in turn the tests that exercise it.

Increase the strictness and coverage of the release test to also
check additional properties:

- that exactly all expected artifacts are published,
  and no unexpected ones are
- that the filename is exactly the right string
- that the binary size and SHA256 checksum matches the one in metadata,
  as well as the .sha256 file
- when applicable, the GPG .asc file is uploaded and its content is
  our fake signature for the right file

Merge signDarwinPKG and signWindowsMSI into one general signArtifact
method since they're exactly the same apart from the build type, and
are now doing more work to compute the SHA256 and size of the signed
artifact.

Also set the SignedPath field (which will be going away when the old
signing code is deleted) and Filename artifact fields.

For golang/go#53632.

Change-Id: Ib7bf831c8f0a0bff14588be432c91d2a5fd1fd5d
Reviewed-on: https://go-review.googlesource.com/c/build/+/447298
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

Also update test data to match, since the fake signing service now
applies a signature. Check that it's a signature of the right type.

For golang/go#53632.

Change-Id: Iea2321e0e0c41ce12e2a6152a2300f03e592c546
Reviewed-on: https://go-review.googlesource.com/c/build/+/447299
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

The default normalization that adjustTar applies is sufficient for
the needs of ConvertPKGToTGZ. There are no additional adjustments
needed since they've already been applied in earlier stages.

This was expected, and a test run hasn't uncovered anything more
to change here, so remove the resolved TODO comment.

For golang/go#53632.

Change-Id: I024bf5fd739ab07200b7f2339f68cfe340a594c3
Reviewed-on: https://go-review.googlesource.com/c/build/+/447959
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>

Delete unused code.

For golang/go#53632.

Change-Id: Ie52e0e067bccb9d363f3b89600bd7fc8521ff332
Reviewed-on: https://go-review.googlesource.com/c/build/+/447276
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>