Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: adopt Go 1.20 as bootstrap toolchain for Go 1.22 #54265

Closed
rsc opened this issue Aug 4, 2022 · 55 comments
Closed

build: adopt Go 1.20 as bootstrap toolchain for Go 1.22 #54265

rsc opened this issue Aug 4, 2022 · 55 comments
Labels
early-in-cycle A change that should be done early in the 3 month dev cycle. Proposal Proposal-Accepted
Milestone

Comments

@rsc
Copy link
Contributor

rsc commented Aug 4, 2022

I propose that starting with the Go 1.22 dev cycle, we require Go 1.20 as the Go bootstrap version (instead of Go 1.17),
and in general that once a year we bump the bootstrap version forward to last year's Go version.

#44505 was about updating the Go bootstrap version from Go 1.4 to Go 1.16 in Go 1.18; we were delayed and ended up updating to Go 1.17 for Go 1.20 instead. Now we have automation in place that makes it easy to update the bootstrap toolchain. To keep ourselves up-to-date and reduce the amount of technical debt we carry forward, I suggest that we establish a pattern where once a year we bump the bootstrap version forward to the previous year's Go release.

With the current release schedule, this would mean:

Go 1.22 (dev starts Aug 2023, release Feb 2024) bootstraps with Go 1.20 (released Feb 2023)
Go 1.24 (dev starts Aug 2024, release Feb 2025) bootstraps with Go 1.22 (released Feb 2024)
...

In the proposal text for #44505, I wrote about the benefits to packagers of keeping the bootstrap release fixed for longer periods of time, perhaps updating every four years. But now I am proposing that we update every one year. Why? A few reasons.

  1. We have seen a few thorny issues, such as runtime: "sweep increased allocation count" on linux/arm64 during bootstrap #42543, caused by bugs in old toolchains being used for bootstrap; updating more frequently eliminates those kinds of bugs entirely, reducing load on the Go team.

  2. It turned out to be a lot more work on our side to update the bootstrap toolchain than I realized when I proposed build: adopt Go 1.17 as bootstrap toolchain for Go 1.20 #44505. Clearly it needed to happen regardless, and we now have good automation in place to make the next one easy. Doing it every year will keep that process working much more smoothly than if we do every four years.

  3. I am not sure that there is a significant burden on packagers if we have a predictable schedule: the work should be to update their bootstrap URLs once a year, perhaps after refactoring to make that an easy operation.

Thoughts? Any reasons the burden on packagers would be significantly higher than I am estimating?

@gopherbot gopherbot added this to the Proposal milestone Aug 4, 2022
@ianlancetaylor
Copy link
Contributor

If we do this I think we should set up a builder that starts with the Go 1.4 sources and builds the current release, whatever it is, without relying on any binary build of Go.

@mdempsky
Copy link
Member

mdempsky commented Aug 4, 2022

Bumping the bootstrap version from Go 1.17 to Go 1.20 will mean the toolchain can start using generics. That's exciting, but also a little scary because generics are still a relatively recent feature. Go 1.20 will be only the 3rd release to support generics; and switching to unified IR is going to mean a largely new implementation of them. We also don't have gofrontend support for generics yet.

I'm in support of setting a tentative plan to bump Go 1.22's bootstrap toolchain to Go 1.20, so that downstream integrators can prepare for that. But I think that after the Go 1.21 release (so after a cycle's worth of Go 1.20 support), it would be good for the team to assess our confidence in depending on that release+generics for bootstrapping.

That concern aside, I think a regular yearly cadence for bumping bootstrap releases sounds appealing. It means whenever we bump the bootstrap toolchain, the new version will still be within the support cycle. So maybe it's viable to actually fix bootstrap toolchain issues, rather than awkwardly skirting around them like we've had to do with Go 1.4. (And if not, the workarounds only need to live for a year anyway.)

@rsc
Copy link
Contributor Author

rsc commented Aug 4, 2022

@ianlancetaylor I am not sure what that builder would be checking, exactly. Suppose it did:

  1. Build Go 1.17 with Go 1.4.
  2. Build Go 1.20 with Go 1.17.
  3. Build the CL under test with Go 1.20.

Steps 1 and 2 will run the same code on every CL. Only step 3 will vary. And that step will match any builder that is using Go 1.20 directly for bootstrap.

@randall77
Copy link
Contributor

@rsc I think that would check that those steps work on the builders.
We would want to check that nothing changed on the builders that would break the bootstrap chain (os version, gcc version, etc.). I don't think we've ever had a problem on Linux, but OSX breaks old versions of Go which prevent that bootstrap chain from working. Running the full bootstrap chain ensures we have a way to purely reconstruct from source on at least one platform.
If it failed, of course, we'd have to issue a new point release of something. Not sure we want to sign up for that or not.

I agree it wouldn't need to run every CL. Periodically, or when we change builder configs would be fine.

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Aug 4, 2022

Right, I didn't mean to start from 1.4 on trybots or on every CL. Sorry for being unclear. I mean an automated test, on the builders, that we can bootstrap from source with nothing but a binary C compiler. It would be fine if we just ran that test before every release.

@rsc
Copy link
Contributor Author

rsc commented Aug 5, 2022

I'm fine with having a test that we occasionally run on some stable, backwards-compatible system like linux/amd64. I'm not sure it accomplishes much now that the build dashboard is pedantic about the specific version of Go being used for bootstrap. (In the past it was kind of random which Go bootstrap different builders had baked in. Now the dashboard is in charge of loading it onto each builder and the code is written so that they all use the specific bootstrap version.)

It is certainly not a goal to have the "start with Go 1.4" sequence work on every supported operating system. Keith pointed out that Go 1.4 binaries don't work on macOS anymore, for example (I told Apple we were fine with that back when the M1 chips came out and had trouble emulating the Go 1.4 x86 binaries correctly), and other ports didn't exist back then. But a script we run occasionally on linux/amd64 sounds fine.

@aclements
Copy link
Member

Summarizing our discussion from today's compiler/runtime meeting:

  • Several of us felt that there isn't a lot of benefit from regularly updating the bootstrap, versus doing it when there's some feature we actually want to take advantage of in the toolchain. @martisch suggested keeping a bootstrap friction issue open where people log friction they run into that would be addressed by a newer bootstrap version, and we periodically evaluate that issue to see if an upgrade is warranted.
  • I just did a small exercise in cmd/dist (CLs not quite ready to mail) where I did some cleanup that the 1.17 bootstrap enabled, and realized 1.19 would enable further cleanup. So I made a build-tagged version that did the 1.19 cleanup so that would happen essentially automatically once we do upgrade again. I came out of that exercise with mixed feelings.
  • @cherrymui pointed out that upgrading the bootstrap regularly means we would always be using a release that's still in the official support window. The long "kind of sort of" supported status of 1.4 was a bit of a mess.

@gopherbot
Copy link

Change https://go.dev/cl/427958 mentions this issue: cmd/dist: simplify exec.Cmd helpers for Go 1.19

@rsc
Copy link
Contributor Author

rsc commented Sep 7, 2022

The main rationale in my view is to do this so we don't forget how to do it. Updating to Go 1.17 was a big ordeal, and if we do it yearly, it will not be.

Note also #49686 which is the kind of thing we avoid by updating quickly (here we haven't updated quickly enough, but same kind of thing).

We also need to get into the habit of filing issues for completing tree-wide updates at the next bump. For example when we introduced strings.Cut we updated all the non-bootstrap-compiled code, but we should also file an issue linked to the next bootstrap update to complete the conversion of the tree then. Similarly things like strings.Builder which is going on now.

@rsc
Copy link
Contributor Author

rsc commented Sep 7, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@mewmew
Copy link
Contributor

mewmew commented Sep 8, 2022

From #54265 (comment):

If we do this I think we should set up a builder that starts with the Go 1.4 sources and builds the current release, whatever it is, without relying on any binary build of Go.

From #54265 (comment):

I'm fine with having a test that we occasionally run on some stable, backwards-compatible system like linux/amd64. I'm not sure it accomplishes much now that the build dashboard is pedantic about the specific version of Go being used for bootstrap.

Being able to do a full source build of Go (e.g. starting from Go 1.4) without having to rely on a binary release would be required for Trusting Trust as @ken illuminated. Having support for only one major operating system/architecture target (e.g. linux/amd64) should be fine, since Go can be cross-compiled for other OS/ARCH targets.

@rsc
Copy link
Contributor Author

rsc commented Sep 8, 2022

See also https://dwheeler.com/trusting-trust/.

@gopherbot
Copy link

Change https://go.dev/cl/430336 mentions this issue: cmd/link/internal/ld, syscall: use libc based msync on darwin for Go ≥ 1.20

gopherbot pushed a commit that referenced this issue Sep 13, 2022
…≥ 1.20

Direct syscalls should no longer be used on darwin. Instead, directly
call libc's msync when using Go ≥ 1.20 for bootstrap.

For #54265

Change-Id: Ie3f1e6ccd1a06e7f0ddd88cdef5067393a69e8db
Reviewed-on: https://go-review.googlesource.com/c/go/+/430336
Reviewed-by: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Tobias Klauser <tobias.klauser@gmail.com>
@gopherbot
Copy link

Change https://go.dev/cl/430496 mentions this issue: cmd, syscall: use syscall.Mmap on solaris for Go ≥ 1.20

gopherbot pushed a commit that referenced this issue Sep 15, 2022
CL 413374 added syscall.Mmap on solaris. Use it in cmd/compile and
cmd/link if the bootstrap toolchain is Go ≥ 1.20.

For #52875
For #54265

Change-Id: I9a0534bf97926eecf0c6f1f9218e855344ba158f
Reviewed-on: https://go-review.googlesource.com/c/go/+/430496
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
gopherbot pushed a commit that referenced this issue Sep 16, 2022
When running on Go 1.19, we can further simplify some of the exec.Cmd
helpers due to API improvements. There's not much point in doing this
while the bootstrap is still 1.17, but this will queue up this
simplification in an obvious way for when we next upgrade the
bootstrap toolchain (#54265).

Updates #44505.

Change-Id: I2ebc3d5c584375ec862a1d48138ab134bd9b2366
Reviewed-on: https://go-review.googlesource.com/c/go/+/427958
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
@rsc
Copy link
Contributor Author

rsc commented Sep 28, 2022

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@06kellyjac
Copy link

Just to clarify is a c build -> go 1.4 -> 1.17 -> 1.20 -> 1.22 bootstrap process included in this proposal (at least for platforms that support it)

Or is this proposal just focused on bootstrapping 1.22 from a 1.20 binary?

@rsc
Copy link
Contributor Author

rsc commented Oct 5, 2022

Yes, at least on a Linux builder we would periodically test the entire bootstrap sequence.
One other detail that has become clear recently is that when we say Go 1.20 is required, it will be the latest point release of Go 1.20 at the time we adopt the requirement, so that any bugs fixed in the first few point releases don't have to be worked around.

It looks like Go 1.20 will require Go 1.17.13, not Go 1.17.

@rsc
Copy link
Contributor Author

rsc commented Oct 6, 2022

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc changed the title proposal: build: adopt Go 1.20 as bootstrap toolchain for Go 1.22 build: adopt Go 1.20 as bootstrap toolchain for Go 1.22 Oct 6, 2022
@rsc rsc modified the milestones: Proposal, Backlog Oct 6, 2022
@mdempsky
Copy link
Member

@pmur FYI (as owner), the linux-ppc64le-power10osu builder appears to still be using Go 1.17, instead of Go 1.20: https://build.golang.org/log/8e73f3aca215a0546421a3bfdb2ca0e1e58905c3

@mdempsky
Copy link
Member

mdempsky commented Jul 20, 2023

@steeve @changkun FYI (as owners), the ios-arm64-corellium builder appears to still be using Go 1.19, instead of Go 1.20: https://build.golang.org/log/5b0b66789350a3e8e83eab8425d0c66ebdbc92df

Edit: Also, android-arm{,64}-corellium is still using Go 1.18: https://build.golang.org/log/449155fd6a03e937b3a68f5d6ddb33363763f4b7

@pmur
Copy link
Contributor

pmur commented Jul 20, 2023

@mdempsky for the previous releases, will the buildlet download the newer 1.20 bootstrap toolchain? If so, I can remove the local bootstrap toolchain now the baseline supports power10.

@dmitshur
Copy link
Contributor

@pmur You'll likely also need to remove the GOROOT_BOOTSTRAP line pointing to /usr/local/go-bootstrap in the x/build/dashboard's host config. After that, yes, the x/build/cmd/buildlet will default to using go1.20.6 as the bootstrap for all branches that builder runs on, and should work okay.

@jefferyto
Copy link

jefferyto commented Jul 21, 2023

Why can't OpenWRT build a bootstrap toolchain for OpenWRT by cross-compiling from another distribution (with CGO_ENABLED=0, or starting with Go 1.21 which builds the toolchain without cgo by default)?

All of OpenWrt is cross-compiled from another distribution. The reasons why we're not doing these at the moment are logistical/ideological/too long and irrelevant to this discussion. Thanks for the suggestions, we might end up doing one of these, or I'll figure something out.

@changkun
Copy link
Member

@mdempsky, thanks for the ping. I will update the builder to use 1.20 over the weekend.

@nmeum
Copy link

nmeum commented Jul 21, 2023

I don't understand why these builds are "cumbersome". Building Go 1.4 -> Go 1.17 -> Go 1.20 -> Go 1.22 an extra iteration of a shell script loop. Adding a few more bytes to the shell script once a year to extend the list of versions to iterate over also does not seem "cumbersome".

@rsc it requires us to ensure that all intermediate versions build on all architectures (riscv64, s390x, ppc64le, i386 …) at all times (with newer musl libc versions etc). In your bootstrap path, for example, Go 1.4 doesn't support riscv64 (which is also why we bootstrap with gccgo presently). This is by no means an academic issue, on these more obscure architectures Go does break occasionally with musl (see #51787 or #58385 for example). The overhead involved with the maintenance of these intermediate versions is what makes this more cumbersome than a simple bootstrap path like gcc → gccgo → Go 1.20.

That being said, I do understand why it is desirable for Go compiler developers to be able to use the new and shiny features provided by newer Go versions, but this certainly makes it more difficult to have a clean bootstrap path if gccgo isn't updated accordingly.

@rsc
Copy link
Contributor Author

rsc commented Jul 23, 2023

@nmeum Thanks. I do see how bootstrapping architectures that post-date those earlier versions of Go would be more difficult. It seems like cross-compiling would be the answer there. The unfortunate fact is that gccgo is likely to lag behind the main Go releases.

@gopherbot
Copy link

Change https://go.dev/cl/512275 mentions this issue: make.bash,bat,rc: use Go 1.20.6 instead of Go 1.17.13

gopherbot pushed a commit that referenced this issue Jul 24, 2023
This was missed in the update of the bootstrap toolchain
and should help people who don't set GOROOT_BOOTSTRAP
and instead assume these scripts will find the right one.

For #54265.

Change-Id: I37a0d0976006d13b73df00013780be5abf202e91
Reviewed-on: https://go-review.googlesource.com/c/go/+/512275
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Russ Cox <rsc@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/512720 mentions this issue: cmd/dist: fix broken link in README

gopherbot pushed a commit that referenced this issue Jul 25, 2023
While here, also update the go15bootstrap link to use the
shorter go.dev domain and https:// prefix for consistency.

For #54265.

Change-Id: I881eeda235589511a93bf47186f91f6c47c12932
Reviewed-on: https://go-review.googlesource.com/c/go/+/512720
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
@mvdan
Copy link
Member

mvdan commented Aug 24, 2023

I see that this issue remains open for the 1.24 milestone, even though as far as I can tell, the work for 1.22 happened a month ago. Is the plan to keep this a "rolling" open issue to repurpose every two Go versions?

@heschi heschi modified the milestones: Go1.24, Go1.22 Aug 24, 2023
@heschi
Copy link
Contributor

heschi commented Aug 24, 2023

Yeah, but on further consideration, I don't want to. We'll track this in our internal release plan. Moved back and closing.

@gopherbot
Copy link

Change https://go.dev/cl/562619 mentions this issue: _content/doc/install: document minumum bootstrap versions

algitbot pushed a commit to alpinelinux/aports that referenced this issue Feb 17, 2024
bell-sw pushed a commit to bell-sw/alpaquita-aports that referenced this issue Feb 20, 2024
[ commit c184b82e4b323135f3e26dd2c263469a21bfa9ed ]

So long and thanks for all the fish.

See golang/go#54265
gopherbot pushed a commit to golang/website that referenced this issue Feb 21, 2024
Fixes golang/go#65654
Updates golang/go#54265
Updates golang/go#44505

Change-Id: Ia1c9b50a2f66b67db43e8dcd21b1002b66cf30fa
Reviewed-on: https://go-review.googlesource.com/c/website/+/562619
Auto-Submit: Robert Findley <rfindley@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Austin Clements <austin@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
early-in-cycle A change that should be done early in the 3 month dev cycle. Proposal Proposal-Accepted
Projects
Status: Accepted
Development

No branches or pull requests