Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/relui: update for new distribution archive builds #58659

Closed
rsc opened this issue Feb 23, 2023 · 12 comments
Closed

x/build/cmd/relui: update for new distribution archive builds #58659

rsc opened this issue Feb 23, 2023 · 12 comments
Assignees
Labels
Builders x/build issues (builders, bots, dashboards) early-in-cycle A change that should be done early in the 3 month dev cycle. NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone

Comments

@rsc
Copy link
Contributor

rsc commented Feb 23, 2023

For #24904, I have mailed CL 470676, which adds a new make.bash -distpack flag that generates distribution archives.

On the release infrastructure side, x/build needs to start using make.bash -distpack in Go 1.21 and later to produce the distribution archives (including the source archives), instead of the code it currently uses to generate them. The -distpack mode also writes the module files that we need to serve for GOTOOLCHAIN support (#57001). Those need to start being served from go.dev/dl like the other archives.

When writing a VERSION file for Go 1.21 or later, the file should be of the form

goX.Y.Z
time 2022-02-23T12:34:56Z

That time stamp is used as the modification time in the archives (for reproducibility). This means something in the x/build metadata will have to record the time to use for each release and then write that time into the VERSION file.

The reproducibility work enables building distribution archives on any kind of machine. In particular it should be possible to build everything on an appropriately trusted+locked down linux/amd64 system. You use GOOS=goos GOARCH=goarch ./make.bash -distpack.

Note that if we move the generation of the linux/arm distribution archives to linux/amd64, we need to set GOARM=6 as well, to mimic the environment on linux-arm-aws. So GOOS=linux GOARCH=arm GOARM=6 ./make.bash -distpack.

It doesn't matter what goos/goarch is used to generate the source archives (you get the same result on every system, as one would expect), but it does need to be generated by make.bash -distpack for reproducibility.

Once the release infrastructure is updated, we should issue a go1.21alpha1 (built from a master branch is fine, no need for a release branch) to exercise the infrastructure and to test that external users can reproduce the archives (#57120).

/cc @golang/release

@rsc rsc added the early-in-cycle A change that should be done early in the 3 month dev cycle. label Feb 23, 2023
@rsc rsc added this to the Go1.21 milestone Feb 23, 2023
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Feb 23, 2023
@thanm thanm added the NeedsFix The path to resolution is known, but the work has not been done. label Feb 23, 2023
@gopherbot
Copy link

Change https://go.dev/cl/470755 mentions this issue: cmd/release: deprecate

@rsc
Copy link
Contributor Author

rsc commented Feb 23, 2023

One question is how to test that things stay reproducible.

One possible answer would be to use the trusted Linux machine as source of truth but then also build everything on Windows and require getting bit-for-bit identical outputs before blessing those files as part of the release. The Windows machine does not have to be as trusted since it is only a sanity check on the trusted builder. The two OSes Windows and Linux should be sufficiently different to serve as a good check.

gopherbot pushed a commit to golang/build that referenced this issue Feb 23, 2023
Relui replaced release since Go 1.18 or so. We kept it around a bit
longer while there were remaining Go+BoringCrypto releases that still
used the release command. Those are no more, so to avoid confusion,
remove cmd/release code given it's no longer maintained or supported.

Updates golang/go#40279.
Updates golang/go#58659.
Fixes golang/go#45893.

Change-Id: Id0d641bee49c9584c52e5616322f0656b89cd851
Reviewed-on: https://go-review.googlesource.com/c/build/+/470755
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
@heschi
Copy link
Contributor

heschi commented Mar 3, 2023

@rsc once the CL is merged we'll prioritize this. Looking forward to it.

@gopherbot
Copy link

Change https://go.dev/cl/478158 mentions this issue: internal/relui: cross-compile Go 1.21 with -distpack

@heschi heschi self-assigned this Mar 28, 2023
@gopherbot
Copy link

Change https://go.dev/cl/482996 mentions this issue: internal/relui: create module files for releases

gopherbot pushed a commit to golang/build that referenced this issue Apr 18, 2023
In 1.21, make.bash has a new argument, -distpack, which builds the
release archives reproducibly from any host architecture. See the
related bug for more details. This converts relui to use -distpack to
build everything.

Mostly that's as simple as adding an alternate build mode for the source
and binary archives. There are/were some wrinkles to work out:
- We need to generate the timestamp at the top level of the workflow so
  that the source consistency check reuses that same timestamp.
- Since we now get a zip out of the Windows build, we comically have to
  convert that back to a tarball to push it to a buildlet. Or at least I
  thought that was better than running unzip.
- I haven't figured out what to do about the module files yet. I think
  I'd prefer to recreate them in relui -- this code is really not happy
  about getting multiple files back from a build step.
- I didn't make a clear distinction between host and target builders in
  the BuildletStep. Maybe I should. But the distpack codepaths are so
  much smaller that it was easy to verify they didn't use the wrong
  config anywhere.

For golang/go#58659.

Change-Id: I4c4cf5b5450046a62d062d7c0bbfe94157ee9446
Reviewed-on: https://go-review.googlesource.com/c/build/+/478158
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Apr 21, 2023
Building on relui's new support for distpack builds, publish the
toolchain module files it generates. This turns out to be quite a
bit more complicated than I had hoped.

First, save the entire distpack archive, not just the binary
distribution. Extract the binary distribution for use in the normal
workflow. Then extract the modules files (really just the .zip, the
.mod and .info are tiny) to a separate artifact. Publish those to the
CDN alongside the normal release artifacts.

For non-distpack builds, copy the dl2mod code in from CL 475920 and
generate the module files as a postprocessing step on the binary
distribution.

The really annoying part is macOS, where we need to distribute the
signed binaries in the module, not the ones we built. For that, take the
module zip generated by distpack and replace the binaries with the ones
from the signed binary distribution. This is a fairly picky process, but
hopefully means we don't risk drifting out of sync with the canonical
distpacks.

Finally, the testing changes: the fake make.bash needs to emulate
distpack too, which wasn't too bad. And then the new requirement to
process the results of the macOS signing process means that the testing
fakes can't just produce cute text files -- they have to contain the
real distribution. Change the fake macOS packaging tools to deal in
prefixed tarballs.

For golang/go#58659.

Change-Id: I9cca8af2cb562fb39ef805a044109877b4ab7fb8
Reviewed-on: https://go-review.googlesource.com/c/build/+/482996
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/487795 mentions this issue: internal/relui: feed only text to sed in test

gopherbot pushed a commit to golang/build that referenced this issue Apr 24, 2023
The intermediate PKG file is no longer a single line of plain text, it
has binary data (a release tarball) appended as of CL 482996. There is
a good chance said binary data will have a byte sequence that can't be
parsed as UTF-8 or some other character encoding without an error. sed
in turn fails with "sed: RE error: illegal byte sequence" on macOS.

Avoid this problem by using sed on the text and not the binary part of
the file.

For golang/go#58659.

Change-Id: If6acfed4c1f41c7ce3366679808614fecb98ec11
Reviewed-on: https://go-review.googlesource.com/c/build/+/487795
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@heschi
Copy link
Contributor

heschi commented Apr 24, 2023

The bulk of this is done; it would be good to do the reproducibility testing described in #58659 (comment).

@heschi
Copy link
Contributor

heschi commented Apr 25, 2023

Oh, I also intended to prefetch the module files from the module proxy to make sure that automatic toolchain updates started working immediately.

@gopherbot
Copy link

Change https://go.dev/cl/489275 mentions this issue: cmd/relui: address performance issues with module zips

gopherbot pushed a commit to golang/build that referenced this issue Apr 26, 2023
Creating the module zips is surprisingly CPU-intensive, and we end up
doing it many times in parallel, resulting in it taking more than 10
minutes to finish and triggering the watchdog timer.

Disable the watchdog for this task, and give relui more CPU so that it's
less of a problem in the first place. (Collateral damage to other tasks
is still a concern.)

For golang/go#58659.

Change-Id: I65b5c47f9fdc011885c0d5f886f4050551d55f61
Reviewed-on: https://go-review.googlesource.com/c/build/+/489275
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/489415 mentions this issue: internal/relui: wait for module files to appear on proxy.golang.org

gopherbot pushed a commit to golang/build that referenced this issue Apr 26, 2023
At some point the modules will be a key part of the release, just like
the website entries are now. Make sure that they're available on
proxy.golang.org before the release is announced.

For golang/go#58659.

Change-Id: I744c067f6efa06f3ca4cdbaf40a351f583331c16
Reviewed-on: https://go-review.googlesource.com/c/build/+/489415
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/490615 mentions this issue: internal/relui: add reproducibility check to distpack builds

@dmitshur dmitshur changed the title x/build: update for new distribution archive builds x/build/cmd/relui: update for new distribution archive builds May 5, 2023
@gopherbot
Copy link

Change https://go.dev/cl/502537 mentions this issue: internal/relui: build all distpacks on both Windows and Linux

gopherbot pushed a commit to golang/build that referenced this issue Jun 13, 2023
Try to shake out any reproducibility bugs by building each distpack
on two different host types.

For golang/go#58659

Change-Id: Ibe858909bf8469b71ba5c25c1588286f459101e8
Reviewed-on: https://go-review.googlesource.com/c/build/+/502537
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) early-in-cycle A change that should be done early in the 3 month dev cycle. NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
Archived in project
Development

No branches or pull requests

5 participants