cmd/go: ethereum build not reproducible #59500
Comments
jrick
changed the title
|
Some additional info: I can reproduce the same build as done by Linux from OpenBSD. The Windows build is the odd one out. |
seankhliao
added
the
NeedsInvestigation
|
can you get the |
|
Sure. The output is the same for both. |
|
Here's an attempt at diffing the binaries. (I removed the $ go tool objdump geth.exe.linux | sed -E 's/(0x|\+)[0-9a-f]+/0xXXXXXX/g' | cut -f1,3,7 > a $ go tool objdump geth.exe.windows | sed -E 's/(0x|\+)[0-9a-f]+/0xXXXXXX/g' | cut -f1,3,7 > b $ diff -u a b diff output: https://gist.github.com/jrick/c3b3b2ad54212a6a634d66997d4d47b3 |
|
The differences in the text compiled for |
|
I am able to reproduce the In order to rule out differences in
|
|
Curiously, building using a Go compiler in a WSL mount of the Windows clone of the repo produces yet a third checksum. 😵💫 |
bcmills
added
the
compiler/runtime
|
Output from Those builds reproduced the @golang/compiler: at first blush this looks like a compiler and/or linker reproducibility bug to me. Note that these are for pure-Go builds (with |
|
I'm not 100% sure I built from Windows properly (getting a gomote to download and build was fiddly), but in my binaries, the first difference is in When built from Windows: When built from Linux: It looks like on Windows build the GOOS and GOARCH checks were deadcode-eliminated, but not on the Linux build? |
|
(That deadcode elimination doesn't make sense to me. GOOS and GOARCH are clearly not compile-time constants, as they are dependent on an environment variable) |
This comment was marked as outdated.
This comment was marked as outdated.
1 similar comment
|
I poked at this a little as well, but I think the difference with the gomote appears to be due to differences in zbootstrap.go: |
This comment was marked as outdated.
This comment was marked as outdated.
|
Maybe this is the culprit here: https://go.googlesource.com/go/+/5e93a2c2042484ee7a941e967294a5248ab6a593/src/cmd/dist/util.go#379 ? |
|
@thanm, none of these examples are on |
I believe this is because |
|
@jrick, users should not need to set any |
|
...but the difference does hold with |
Executable does contain a copy of the "goarm" function, even if it is not used/relevant. Linking without "-s -w": When zbootstrap.go contains "defaultGoArm = |
|
I still don't think that explains it? I don't see a difference in bryan@guthrie:~/go$ diff -r ./src /mnt/c/Users/bryan/src/go/src
diff '--color=auto' -r ./src/cmd/cgo/zdefaultcc.go /mnt/c/Users/bryan/src/go/src/cmd/cgo/zdefaultcc.go
18c18,22
< return "c++"
---
> switch goos {
> case "darwin", "ios", "freebsd", "openbsd":
> return "clang++"
> }
> return "g++"
diff '--color=auto' -r ./src/cmd/go/internal/cfg/zdefaultcc.go /mnt/c/Users/bryan/src/go/src/cmd/go/internal/cfg/zdefaultcc.go
18c18,22
< return "c++"
---
> switch goos {
> case "darwin", "ios", "freebsd", "openbsd":
> return "clang++"
> }
> return "g++"
Only in /mnt/c/Users/bryan/src/go/src/cmd: .vscode
diff '--color=auto' -r ./src/go/build/zcgo.go /mnt/c/Users/bryan/src/go/src/go/build/zcgo.go
5c5
< const defaultCGO_ENABLED = ""
---
> const defaultCGO_ENABLED = "0" |
|
I agree that with |
Strange. Perhaps this is some sort of artifact of how gomotes work? What is your "defaultGoArm" setting, just out of curiosity? |
My const defaultGOARM = `7`But IIUC the |
|
Under what circumstances do we rewrite Edit: Ignore above, that is because this behavior has changed since 1.20 in https://go.dev/cl/470695. |
|
Thanks for that (CL 470695). Yeah, building on linux from 1.20 release branch I get "defaultGOARM" of 5. |
|
The closest difference i can find is in |
bcmills
added
the
compiler/runtime
|
The goarm-related inconsistencies are definitely present in Go 1.20 and fixed for Go 1.21 already. |
|
Worth pointing to my comment #59525 (comment), which applies here too. cmd/go imports go/build, and it reproduces with the setup described there. The original report is because of a bug in Go 1.20, which has been fixed. Reports of problems in the current devel tree are likely to turn out to be caused by other incidentals, such as doing this from a git checkout (as opposed to a clean source tree, because of embedded VCS information) or accidentally leaving a few files behind in one computer versus the other. By far the best way to set up the test environment is to make a tgz of your source tree (or use ./make.bash -distpack to get one) and then copy it to the multiple test machines, unpack in a clean directory, and ./make.bash from there. It is true that |
|
So coming from #59525, I am now running a development version of Go with VERSION file. |
|
Does |
|
If the two do not reproduce, then the next thing to check would be any Go-relevant environment variables (like CGO_ENABLED or GOARM) being set or unset differently on the two machines. GOOS, GOARCH, and GOROOT are okay to be different, of course. |
|
Building master (at 1635205) without the CL shows different hashes for all of the dist files. |
|
Well I figured out why that was. I had a stray |
|
What about geth.exe? I was able to build it reproducibly for windows/amd64 on windows/amd64 and linux/amd64 machines. I used
|
|
Nope, still getting different geth.exe builds with that exact command. The diff looks the same as #59500 (comment). |
|
Now that the src archives match, the next thing to check is Those need to print all the same hashes. Once those match, we can move on to Ethereum differences. I am building from a copy of go-ethereum I built using git archive and then extracted into a clean directory, so there's no .git during the build. Using the same tgz on both systems also helps me make sure the source code tree really is the same in both places. I was using specifically https://swtch.com/tmp/eth59500.tgz. |
|
(If you can get a reproducible build from the tgz but not from a checkout then maybe you need -buildvcs=false or maybe something else needs more examination. But trying the same tgz on both machines is probably the right next step.) |
|
All of the distpack hashes are matching when I build for windows from linux. I used this VERSION: with Go master checked out at 4b154e5. But using that .tgz, I'm still getting different geth.exe hashes: |
|
I'm seeing basically the same results as @jrick. The linux hash of From linux: From windows (under msys2): Then building |
|
I have reproduced a non-reproducibility problem building geth. Will investigate. |
|
I used separate source trees, but if I build inside WSL from the same source tree that I used for the native windows build (building in /mnt/c/...), I get the same hash as from the source tree on the Linux FS. |
rsc
changed the title
dist cause non-reproducible builds for binaries that import go/build
|
Possibly related to #59571 |
|
Yes, urfave/cli is the root cause for geth too. |
|
Once #59571 is fixed, all evidence points to this being fixed too. We can leave it open until then and see. |
|
I am able to build geth reproducibly using the same source checkout (the extracted tgz), with the Linux test using the same files as Windows through the WSL /mnt/c/ path. However, I have not been able to create a reproducible geth.exe when building from two different go-ethereum git checkouts at same commit, with clean source trees. I double checked my Git for Windows settings, and autocrlf is false. If I do the Linux build from the same Git source tree as windows, that ends up being reproducible. So I checked the sha256 hashes of the Go source files in the go-ethereum checkouts and they are all different. 😕 Not sure what is going on with that, but seems like the Go reproducibility issues are solved. |
|
Seems to be the line endings windows: linux: |
|
Amazing how CRLFs are still a problem in 2023, but I'm glad you solved everything else. Thanks for confirming. |
During bootstrapping, cmd/dist writes a file indicating which GOOS/GOARCH combinations are valid, and which support cgo-enabled builds. That information previously went into the go/build package, but today it fits in more naturally in the internal/platform package (which already has a number of functions indicating feature support for GOOS/GOARCH combinations). Moreover, as of CL 450739 the cmd/go logic for determining whether to use cgo is somewhat more nuanced than the go/build logic: cmd/go checks for the presence of a C compiler, whereas go/build does not (mostly because it determines its configuration at package-init time, and checking $PATH for a C compiler is somewhat expensive). To simplify this situation, this change: - consolidates the “cgo supported” check in internal/platform (alongside many other platform-support checks) instead of making it a one-off in go/build, - and updates a couple of tests to use testenv.HasCGO instead of build.Default.CgoEnabled to decide whether to test a cgo-specific behavior. For #58884. For #59500. Change-Id: I0bb2502dba4545a3d98c9e915727382ce536a0f3 Reviewed-on: https://go-review.googlesource.com/c/go/+/483695 Auto-Submit: Bryan Mills <bcmills@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com>
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go env (windows)Outputgo env (linux)OutputWhat did you do?
Performing a cross compile, that should be bit-for-bit reproducible, of the geth executable from the go-ethereum repo checked out at commit b946b7a13b749c99979e312c83dce34cac8dd7b1 results in different file hashes when building on Linux vs Windows.
What did you expect to see?
Same executable built form both hosts. These are the same procedures we use for reproducible cross compiles of other packages, and have not had an issue until building this project.
What did you see instead?
Differing builds
The text was updated successfully, but these errors were encountered: