Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/internal/task: provide a release archive with all the syso files, or release the files separately #54174

Closed
blorente opened this issue Aug 1, 2022 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@blorente
Copy link

blorente commented Aug 1, 2022

What version of Go are you using (go version)?

1.17 SDK, via Bazel's rules_go: https://github.com/bazelbuild/rules_go.

Does this issue reproduce with the latest release?

Yes, the latest release (1.18.4 from go.dev/dl) has that issue.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/blorente/Library/Caches/go-build"
GOENV="/Users/blorente/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/blorente/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/blorente/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.18.4/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.18.4/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.18.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/0b/d4kmqgkx0w5bs1qb8cktwfpw0000gn/T/go-build1763548739=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I tried to build a binary for linux using the musl C toolchain.

rules_go tries to build the standard library by downloading a copy of the sdk in the host architecture (e.g., it would be https://go.dev/dl/go1.18.4.darwin-amd64.tar.gz for my architecture), and then doing roughly:

  • Create a new dir, let's call it foo.
  • Copy src and pkg there.
  • Set foo as the new GOROOT, and set GOOS and GOARCH to the target platform (in this case, GOOS=linux, GOARCH=x86_64
  • Build the standard library by calling go install.

This builds correctly with musl.

What did you expect to see?

However, if I try to link a binary that uses the race detector to this standard library, it will fail to find libc-specific symbols from the thread sanitiser. This issue has been solved in the precompiled syso files for race, in src/runtime/race.

And, in fact, if reproduce the process but manually place race_{GOOS}_{GOARCH}.syso in src/runtime/race, the syso is packed into the race.a archive, and the binary links successfully.
Since the go sdk is supposed to cross-compile, I would have expected this operation to succeed.

What did you see instead?

Since this tool strips the syso for platforms other than the host, the syso can't end up in the .a archive for the package.

Further info

I'm preparing to contribute a fix to rules_go, which involves getting a hold of the sysos somehow. My question is:

What would be the recommended way to get those syso files?

Should I download the src sdk (e.g. https://go.dev/dl/go1.18.4.src.tar.gz) and incur the bandwidth penalty? Should I download the binaries directly from the GitHub mirror, or should I do something else?

@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Aug 1, 2022
@gopherbot gopherbot added this to the Unreleased milestone Aug 1, 2022
@dmitshur
Copy link
Contributor

dmitshur commented Aug 1, 2022

CC @golang/release.

@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 1, 2022
@dmitshur dmitshur changed the title x/build/internal/task: Release a version of the sdk with all the syso files, or release the files separately. x/build/internal/task: provide a release archive with all the syso files, or release the files separately Sep 7, 2023
@dmitshur
Copy link
Contributor

dmitshur commented Sep 7, 2023

The decision to not ship race detector syso files for other platforms was made in 2018 in cmd/release (CL 144281), to reduce release archive size. It was ported to cmd/relui and still applies to Go 1.20.x releases:

$ ls -1 ./go1.20.8.src/src/runtime/race/*.syso
./go1.20.8.src/src/runtime/race/race_darwin_arm64.syso
./go1.20.8.src/src/runtime/race/race_linux_arm64.syso
./go1.20.8.src/src/runtime/race/race_linux_ppc64le.syso
./go1.20.8.src/src/runtime/race/race_linux_s390x.syso

$ ls -1 ./go1.20.8.darwin-arm64/src/runtime/race/*.syso
./go1.20.8.darwin-arm64/src/runtime/race/race_darwin_arm64.syso

Starting with Go 1.21, the decision of which files to include in release archives lives in cmd/distpack (see https://go.dev/blog/rebuild for more context), and distpack did not carry over the previous dropUnwantedSysos behavior:

$ ls -1 ./go1.21.1.darwin-arm64/src/runtime/race/*.syso
./go1.21.1.darwin-arm64/src/runtime/race/race_darwin_arm64.syso
./go1.21.1.darwin-arm64/src/runtime/race/race_linux_arm64.syso
./go1.21.1.darwin-arm64/src/runtime/race/race_linux_ppc64le.syso
./go1.21.1.darwin-arm64/src/runtime/race/race_linux_s390x.syso

So it seems there's nothing more to do in this issue. @blorente Is the above missing anything?

@heschi heschi added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 19, 2023
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 19, 2023
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Oct 19, 2023
@blorente
Copy link
Author

blorente commented Nov 7, 2023

Sorry not answering before. There is indeed no more to do, thanks for the investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Archived in project
Development

No branches or pull requests

4 participants