Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: build cross-compiled releases for secondary ports #53862

Closed
ianlancetaylor opened this issue Jul 14, 2022 · 5 comments
Closed

x/build: build cross-compiled releases for secondary ports #53862

ianlancetaylor opened this issue Jul 14, 2022 · 5 comments
Assignees
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@ianlancetaylor
Copy link
Contributor

Per the discussion in #53383 we should develop a way to build a cross-compiled Go release, without requiring a C cross-compiler, but such that the release will itself default to using cgo as usual. This means that the first time that this new release is used, it will invoke the system C compiler to build the packages that require cgo: the net, os/user, and runtime/cgo packages. Those packages, built with cgo, will thereafter live in the build cache.

We will then be able to use this mechanism as part of our release process to automatically, easily, and safely build releases for a range for second ports. For first-class ports I expect that we will continue to build the releases on the systems themselves with full cgo support included. But for secondary ports we don't want to rely on having a builder for the approach system. A cross-compiled release will be easy for our users to download and use.

A disadvantage of this is that the cmd/go program included in the cross-compiled binary release will be built without cgo, meaning that it will only use the Go DNS resolver. This is mostly fine, and for cases where it is not fine people can fix the problem by running go install cmd/go. However, this will require documentation.

The actual implementation of this may be very similar to the existing bootstrap.bash script, but with some tweaks to ensure that the generated release will default to using cgo.

Of course for secondary ports that do not support cgo (currently linux/ppc64, js/wasm, plan9/*, windows/arm) the generated binary release should not expect to use cgo by default.

@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 14, 2022
@ianlancetaylor ianlancetaylor added this to the Backlog milestone Jul 14, 2022
@rsc
Copy link
Contributor

rsc commented Nov 8, 2022

Looked into this. It works already. Just cross-compile a toolchain with CGO_ENABLED="" ./make.bash.

Specifically, this script worked fine for cross-compiling from my Mac to build a Linux distribution.
When I ran 'go env' in the Linux distribution, it properly defaulted to using cgo.

hostos=darwin
hostarch=amd64
targetos=linux
targetarch=amd64

set -e
rm -rf bin pkg
cd src
CGO_ENABLED="" GOOS=$targetos GOARCH=$targetarch ./make.bash
cd ..
mv bin/${targetos}_${targetarch}/* bin  # move target binaries to right place
rmdir bin/${targetos}_${targetarch}  # remove old place
rm -rf pkg/${targetos}_${targetarch}  # remove target archives (they will be rebuilt)
rm -rf pkg/tool/${hostos}_${hostarch}  # remove host tools

x/build/internal/task/buildrelease will have to be updated to know how to do these steps for cross-compiled toolchains, but there are no changes necessary in cmd/dist to enable that.

Leaving for @heschi and @golang/release to possibly repurpose this issue for updating buildrelease.

@rsc rsc changed the title cmd/dist: provide a way to build a cross-compiled toolchain that will use cgo x/build: build cross-compiled releases for secondary ports Nov 8, 2022
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 8, 2022
@rsc
Copy link
Contributor

rsc commented Nov 8, 2022

To be clear, the cross-compiled Linux distribution in the example above differs from the standard one in that it requires a C compiler to build programs using the standard library (for example package net) when cgo is enabled. So we shouldn't, say, cross-compile all our ports from macOS machines. But cross-compiling all the secondary ports from linux/amd64 should be fine.

@rsc
Copy link
Contributor

rsc commented Nov 8, 2022

Refining my previous comment, today the Windows port does not use cgo for any of std (except runtime/cgo, which is the dependency implicitly added to all cgo packages):

% CGO_ENABLED=1 GOOS=windows go list -f '{{if .CgoFiles}}{{.ImportPath}}{{end}}' std
runtime/cgo
% 

So even the first-class Windows ports could reasonably be cross-compiled from a linux/amd64 system. If we had a Linux program to generate the .msi files for us too, then we would not need Windows machines anymore for the actual generation of releases, which reduces the number of distinct trusted machines to juggle. (We'd probably want to arrange a test that copied the built release over to a Windows machine and ran ./run.bash --no-rebuild, but that doesn't require the machine to be as trusted as building the release.)

For the first-class macOS ports, there are still a few cgo packages remaining even though net is gone:

% CGO_ENABLED=1 GOOS=darwin go list -f '{{if .CgoFiles}}{{.ImportPath}}{{end}}' std
os/signal/internal/pty
os/user
plugin
runtime/cgo
% 

We could plausibly remove the macOS use of cgo in os/signal/internal/pty, os/user, and plugin too. Then the macOS distributions could also be built from linux/amd64, assuming a Linux program to generate the .pkg files for us.

This issue is only about building secondary port distributions, but we should keep in mind the possibility of using that same mechanism for building non-Linux primary port distributions.

@heschi
Copy link
Contributor

heschi commented Mar 23, 2023

I'm making active progress on this.

@heschi
Copy link
Contributor

heschi commented May 2, 2023

This is done as part of the work for #58659. Starting in 1.21 we will publish binaries for all ports.

@heschi heschi closed this as completed May 2, 2023
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 2, 2023
@dmitshur dmitshur modified the milestones: Backlog, Unreleased, Go1.21 May 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Archived in project
Development

No branches or pull requests

6 participants