The concept of a scratch directory has been ad-hoc, making for redundant
and somewhat confusing code. Create a ScratchFS type which encapsulates
the logic. We now just pass bare filenames around, and ScratchFS handles
adding the workflow ID prefix.

For golang/go#63147

Change-Id: Ia2a79aae2cfb2160c0b21f71d1347b8f26269885
Reviewed-on: https://go-review.googlesource.com/c/build/+/529295
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

We're seeing a proliferation of tasks that want to run a command or two
on an arbitrary worker machine. Using buildlets for that presents two
problems: it's an extremely awkward API, and we're planning to get rid
of them not so long from now.

Add support for running an arbitrary script to our Cloud Build wrapper
and switch all such tasks over to it. Next step will be the distpack
builds.

The RunScript API may be a bit overspecialized to the current use cases,
but it's nice to be able to replace a bunch of ceremony with a single
call. If we need something more flexible in the future we can do it
then.

For golang/go#63147

Change-Id: Iefcfae942c75816b8a49b1bd8ca06377f7246267
Reviewed-on: https://go-review.googlesource.com/c/build/+/529296
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

Now that we have Cloud Build support, we can also use it in the Go
release process.

A few notes:
- We now use the latest Go release to bootstrap. I think this is fine?
- We still pass a tar of all the distpacks around. It might be possible
  to improve that, since GCB is capable of uploading multiple files
  easily enough, but I didn't want to make too many changes all at once.
- GCB needs to be able to read the relui scratch space, and write to its
  own.

We still need Windows buildlets to build MSIs and do the
reproducibility check, and Macs to build and extract packages. It might
be possible to do the packaging steps on Linux, but the reproducibility
check is pointless unless it really runs on something different. I
suspect we'll need to use the Swarming API for lack of other options.

For golang/go#63147

Change-Id: Ib4e73cadf314512460e7e04ff2a107c91e52516a
Reviewed-on: https://go-review.googlesource.com/c/build/+/529297
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

A few small fixes:

Set the service account on the build.

Fix the paths in the x509bundle task.

When Cloud Build saves artifacts, it does
gsutil cp <output file> <output path>, which has the effect of losing
all the directory structure. We have places that need the structure,
most notably the dependency update part of the tagging process, which
works with multiple go.mod files in different directories. Fortunately
it was very easy to reimplement.

For golang/go#63147

Change-Id: If210e0cfe205305a84d068cdc404f38e44016fb5
Reviewed-on: https://go-review.googlesource.com/c/build/+/530335
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

I completely misunderstood/misremembered how the signing service worked,
and convinced myself that the signed results were written back to
relui's scratch space, next to the input artifacts. That was totally
wrong: they're written to a per-signing-run scratch directory that we
put under the top-level scratch dir. So it was just broken.

Unfortunately, the fake signing service agreed with me! So all the tests
passed. Even more unfortunately, the production signing service produces
outputs with the same base name as the inputs, so when my code went to
look for the signed artifacts, it found the unsigned ones. My dry run
workflows appeared to succeed, but were going to ship unsigned binaries.

But I got lucky: the GPG signing process produces totally new files,
and reading those from the wrong directory failed. I didn't notice that
because I always cancelled the dry runs before the 3-hour Mac signing
process finished.

This CL changes the signing service to behave much more realistically,
adds the signing output directory as a first-class concept to the
release workflow, and fixes the bug.

For golang/go#63147.

Change-Id: Iba4c54ff2277441d2a34859e166767af1ef599ba
Reviewed-on: https://go-review.googlesource.com/c/build/+/531977
Auto-Submit: Heschi Kreinick <heschi@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>

Expansions were kind of fun for the smaller workflows, but had a major
flaw: there was no way to get data out of an expansion. Put another way,
once a workflow entered an expansion it basically couldn't leave.
That was okay as long as the larger workflows were static, but we're
about to switch to LUCI for testing, and at that point we can't compile
in the list of builders. I suppose we could read them in at startup,
but I'd like to do less of that kind of thing, not more.

Change them to allow returning a single Value. That might prove
limiting, but I'd really prefer not to have a combinatorial explosion
of expansion API just yet.

Unfortunately, I didn't see an easy way to allow this and still do
validation that there aren't orphaned tasks. In practice, even the most
trivial test should catch the problem, so meh. (We didn't used to have
stall detection, at which point it mattered much more.)

For golang/go#63147

Change-Id: I855523a72b87b0ed8089066f246e5869f28e6c89
Reviewed-on: https://go-review.googlesource.com/c/build/+/533860
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>

CL 534156 was written under the assumption that the Git checkout
went into a subdir of the workspace. I'm not sure if I somehow was wrong
about that, or if it's changed, but either way it's not the case now. So
do the Git work ourselves and make it work the way we want. Also fix the
output URLs, which were ending up looking like scratchdir/./go.mod. On a
real filesystem that'd be collapsed, but not on GCS.

This works in prod for x repo tagging and x509roots, which are the two
cases where we use Git checkouts I could test easily.

For golang/go#63147

Change-Id: I8a529b624d5c218a2940806fa6472149b4c82c9c
Reviewed-on: https://go-review.googlesource.com/c/build/+/539479
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>

Attempt to normalize the way we access Gerrit.
- Bundle both the Gitiles and Gerrit URLs into the GerritClient.
- Stop baking the project name into the Gitiles URL, it's confusing.
  Make it a property of the tasks instead.
- Use an API on the GerritClient to get archive URLs. NOTE: this goes
  away with the next CL. Sorry.

For golang/go#63147.

Change-Id: Ibc3549b2a9a26c6f50b01de66c7481a69293d239
Reviewed-on: https://go-review.googlesource.com/c/build/+/541378
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

Run all the builds we have available in security-try as part of the
release workflow.

Overall I think this a reasonable approach. The approval process is
smooth now, and really there's no strong justification for not letting
people exercise judgment for which builders are "required".

One small change is that we're testing based off the source archive, not
the binary archive. Since we were already doing a full make.bash run,
we weren't getting much benefit from that anyway. And distpack builds
are fully reproducible, so it literally shouldn't matter after 1.21.

For golang/go#63147.

Change-Id: If8e1093951e2eadf9a813c71926958733b4958fb
Reviewed-on: https://go-review.googlesource.com/c/build/+/541379
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

Give up on distinguishing between infrastructure failures and test
failures. At the end of the day someone has to look at the errors either
way.

At the same time, don't treat context errors as successes. We can't
afford to assume that timeouts and stuff aren't related to the tests
we're running. Even if we could, we'd be missing coverage.

For golang/go#63147.
For golang/go#60443.

Change-Id: I5fe78616b4c8f85b6b1e901b509d291884cf9a27
Reviewed-on: https://go-review.googlesource.com/c/build/+/541380
Auto-Submit: Heschi Kreinick <heschi@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

This has been bugging me and I keep not getting to it.

For golang/go#63147.

Change-Id: I3c6c6a171fdb5d99887738f561daeaf70a0de975
Reviewed-on: https://go-review.googlesource.com/c/build/+/542655
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

Every time I try a new API with the internal project I find that it
doesn't escape the path properly. Try to get it right everywhere.

For golang/go#63147.

Change-Id: I16b4bd32f2b014208e5f058028d7212e8f436c5c
Reviewed-on: https://go-review.googlesource.com/c/build/+/542656
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>

We don't want automated processes to depend on builders we don't control
and therefore can't completely rely on. Export a property, absent from
the proto, that says whether a builder is operated by Google.

For golang/go#63147.

Change-Id: I7bcef1b163c22a4fbd7d5ffd23235369754697fc
Reviewed-on: https://go-review.googlesource.com/c/build/+/544255
TryBot-Bypass: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>

One less thing depending on the old stuff.

The conversion is pretty straightforward. The greenness check selects a
commit at the beginning. Any builders that we want that haven't passed
get rerun with a high priority. If any of those fail, the overall
attempt fails and restarts, at which point it re-selects the commit it's
looking at. That minimizes redundant work, should get a result
relatively quickly, but still allows us to recover from a genuinely
broken build.

Note: prioritization doesn't work quite right yet because golangbuild
doesn't propagate it down to child builds. That's a separate issue.

For golang/go#63147.

Change-Id: I8db7c5a9dfcfef709915f74b7d9d0d4a4edc89bb
Reviewed-on: https://go-review.googlesource.com/c/build/+/544275
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

It seems this was missed in the initial implementation of expansions,
but we never noticed until CL 541379 started using them for advisory
builders in minor release workflows.

For golang/go#63147.

Change-Id: I528a032a581790dbe27a01f2b245383b41215a05
Reviewed-on: https://go-review.googlesource.com/c/build/+/546235
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

These are similar to the existing macOS and Windows signing build types,
but will be used specifically for installer construction only.

For golang/go#63147.

Change-Id: If835cb66024f3119b69e35593c323dbba6e5f5b2
Reviewed-on: https://go-review.googlesource.com/c/build/+/550320
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

The logic is taken from the existing code for constructing Go installers
in the internal/task and internal/task/releaselet packages. It now uses
a common interface for both, and has tests that can be used during local
development of said installers.

Neither of the packages use cgo now, and probably won't need to.
Use the '!cgo' build constraints to make cgo use an intentional
decision rather than accidental.

For golang/go#63147.

Change-Id: Id5aaa3503a66d5bbb97721e284ce0eaef1a63574
Reviewed-on: https://go-review.googlesource.com/c/build/+/550321
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>

ConstructInstaller already changes working directory to workDir, so it
shouldn't be used again in relative paths. This wasn't spotted earlier
because it happens not to be an issue when workDir is an absolute path,
as was produced by t.TempDir().

Also apply some code simplifications, consistency improvements,
and dead code removal.

Even though the installer packages don't need cgo, they'll be imported
elsewhere where cgo is possibly enabled, so it's not viable to use the
!cgo build constraints. Drop them for now.

For golang/go#63147.

Change-Id: Id60c05fa67988e7548b70a55e712dbabfb7d85d7
Reviewed-on: https://go-review.googlesource.com/c/build/+/554057
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>

CL 554057 updated the splitVersion implementation to account for major
Go releases now having a trailing ".0" component, so "go1.34" would be
"go1.34.0" instead. However, our pre-release versions still need to be
handled as before, which that CL inadvertently broke, and no test case
caught.

Update the code, test cases, and add error reporting accordingly.

For golang/go#63147.

Change-Id: Iab49ee70f17930cc2e6b871311c5d32cd68704d2
Reviewed-on: https://go-review.googlesource.com/c/build/+/555315
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

Move installer construction to happen in the same place that installer
signing already happens, under the ConstructInstallerOnly build types
that CL 550320 added.

Reuse the minimal xar parser that Russ implemented in cmd/gorebuild for
extracting signed binaries (that need to be inserted into the .tar.gz)
from a signed .pkg installer. This lets us drop the Xcode dependency
for that task and have it run as part of relui on secured machines
used only for release orchestration.

Delete the unused previous code and supporting files for building
installers, since what's used now is in the internal/installer
packages that CL 550321 added.

For golang/go#63147.

Change-Id: If8b207b7e3739052bc6d5f8ac13bbe5a05b50e0c
Cq-Include-Trybots: luci.golang.try:x_build-gotip-linux-amd64-longtest-race
Reviewed-on: https://go-review.googlesource.com/c/build/+/552016
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

We never build distpacks using buildlets by now. Delete unused code.

For golang/go#63147.

Change-Id: I4a6d2f37d92c16b0eb3338d70197a3e28f681046
Reviewed-on: https://go-review.googlesource.com/c/build/+/552155
Reviewed-by: Carlos Amedee <carlos@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

Go 1.20 has become unsupported, so relui will always take the distpack
code path for all future Go releases. Delete the non-distpack code.

For golang/go#63147.

Change-Id: If2c38cbe6aee21d7a8f5077162e48c180c6b7db0
Reviewed-on: https://go-review.googlesource.com/c/build/+/564218
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>