Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/link: reproducible builds don't work on macOS #40979

Closed
mandrigin opened this issue Aug 22, 2020 · 13 comments
Closed

cmd/link: reproducible builds don't work on macOS #40979

mandrigin opened this issue Aug 22, 2020 · 13 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Go1.16

Comments

@mandrigin
Copy link

mandrigin commented Aug 22, 2020 •
edited

What version of Go are you using (go version)?

$ go version go1.15 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/mandrigin/Library/Caches/go-build"
GOENV="/Users/mandrigin/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/mandrigin/golang/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/mandrigin/golang"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/mandrigin/Desktop/repro-builds-issue/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/y5/rmr800b53s53zt42d2tfmpdc0000gn/T/go-build570372631=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I've created a repository with a small app where I can reproduce this issue easily. It uses one module with native files (go-duktape): https://github.com/mandrigin/golang-reproducible-builds-issue-osx/

The app builds the same source twice into 2 different files
go build -asmflags -trimpath -ldflags=-buildid= -o t1 .
go build -asmflags -trimpath -ldflags=-buildid= -o t2 .

Then it takes sha sums of both files and prints them. Also it prints a diff between these files.

What did you expect to see?

Reproducible builds on both Alpine (in Docker) and macOS.

What did you see instead?

Reproducible builds on Alpine work. (hashes match, no diff between binary files).

Reproducible builds on macOS are broken.
Diff on macOS contains entries like this one

< 005b1850: 6e2f 542f 676f 2d6c 696e 6b2d 3732 3938  n/T/go-link-7298
< 005b1860: 3738 3633 362f 3030 3030 3237 2e6f 005f  78636/000027.o._
---
> 005b1850: 6e2f 542f 676f 2d6c 696e 6b2d 3532 3038  n/T/go-link-5208
> 005b1860: 3330 3433 312f 3030 3030 3237 2e6f 005f  30431/000027.o._

Clang Version

Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
@ianlancetaylor ianlancetaylor changed the title CGO: reproducible builds don't work on macOS cmd/link: reproducible builds don't work on macOS Aug 22, 2020
@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 22, 2020
@ianlancetaylor ianlancetaylor added this to the Go1.16 milestone Aug 22, 2020
@ianlancetaylor
Copy link
Contributor

CC @thanm @neild

@cherrymui
Copy link
Member

It looks like temporary file names are built into the binary. I'll take a look.

@cherrymui
Copy link
Member

Yeah, it seems the case for any cgo program on darwin, since at least Go 1.11 (Go 1.10 doesn't work on my machine).

@cherrymui
Copy link
Member

I think it is the darwin linker adds them. I don't know how to make it not to...

@thanm
Copy link
Contributor

thanm commented Aug 23, 2020

Possibly DWARF related? What happens when "-w" is added to linker options?

@cherrymui
Copy link
Member

-w doesn't help.

If I understand the otool dump correctly, the file paths are in __LINKEDIT segment.

@ianlancetaylor
Copy link
Contributor

Are the paths coming from the command line? Would it help to chdir to the directory, and just pass plain .o files to the linker?

@cherrymui
Copy link
Member

Good point. The command line does include the paths. But if I cd into the temp dir and run the command with only the file name without the directory, the binary still contains the paths...

@ianlancetaylor
Copy link
Contributor

If you chdir, does it record the paths for each object, or is it recording the current directory somewhere?

@cherrymui
Copy link
Member

The paths for each object.

@ianlancetaylor
Copy link
Contributor

Well that's quite annoying.

@gopherbot
Copy link

Change https://golang.org/cl/250944 mentions this issue: cmd/link: strip STAB (symbolic debugging) symbols on darwin

@gopherbot
Copy link

Change https://golang.org/cl/253157 mentions this issue: cmd/link: add a test for reproducible build

@Roasbeef Roasbeef mentioned this issue Oct 1, 2020
Closed
@golang golang locked and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Go1.16
Development

No branches or pull requests
5 participants
@mandrigin @ianlancetaylor @gopherbot @thanm @cherrymui