Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: macOS plugin builds are not reproducible #58557

Closed
misha-ridge opened this issue Feb 16, 2023 · 5 comments
Closed

cmd/go: macOS plugin builds are not reproducible #58557

misha-ridge opened this issue Feb 16, 2023 · 5 comments
Assignees
@matloob
Labels
GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done. OS-Darwin
Milestone
Go1.21

Comments

@misha-ridge
Copy link

What version of Go are you using (go version)?

Tested with Go 1.19 and Go 1.20

$ go version
go version go1.19.5 darwin/arm64

$ go version
go version go1.20.1 darwin/arm64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/roman/Library/Caches/go-build"
GOENV="/Users/roman/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/roman/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/roman/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org/,direct"
GOROOT="/Users/roman/Work/Ridge/tectonic/_deps.darwin-arm64/go-1.20.1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/roman/Work/Ridge/tectonic/_deps.darwin-arm64/go-1.20.1/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/roman/Work/Ridge/tectonic/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/w2/fyw22w5x5737wfqnpw5ngnc40000gn/T/go-build2948842969=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I have created a package with single main function and built it on Mac twice as a plugin:

package main
func main() {}

go build -trimpath -buildvcs=false -buildmode=plugin -o a.so .
go build -trimpath -buildvcs=false -buildmode=plugin -o b.so .

The resulting binaries differ.

Reproducer is at https://github.com/misha-ridge/test-plugin-build

What did you expect to see?

I expected the binaries to be identical.

What did you see instead?

Plugin embeds a temporary directory path, plus some other differences that might be related to it:

--- /dev/fd/63	2023-02-16 10:47:27
+++ /dev/fd/62	2023-02-16 10:47:27
@@ -113,7 +113,7 @@
 00000700  64 65 72 73 2f 39 35 2f  76 7a 77 63 76 35 79 64  |ders/95/vzwcv5yd|
 00000710  33 32 78 33 36 39 7a 30  63 39 74 34 62 66 72 30  |32x369z0c9t4bfr0|
 00000720  30 30 30 30 67 6e 2f 54  2f 67 6f 2d 62 75 69 6c  |0000gn/T/go-buil|
-00000730  64 31 38 39 36 31 38 31  30 36 30 2f 62 30 30 31  |d1896181060/b001|
+00000730  64 34 30 38 38 30 39 37  33 31 37 2f 62 30 30 31  |d4088097317/b001|
 00000740  2f 65 78 65 2f 61 2e 6f  75 74 2e 73 6f 00 00 00  |/exe/a.out.so...|
 00000750  34 00 00 80 10 00 00 00  00 00 11 00 48 2d 00 00  |4...........H-..|
 00000760  33 00 00 80 10 00 00 00  48 2d 11 00 08 2d 00 00  |3.......H-...-..|
@@ -124,7 +124,7 @@
 000007b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 000007c0  28 11 12 00 44 01 00 00  00 00 00 00 00 00 00 00  |(...D...........|
 000007d0  00 00 00 00 00 00 00 00  1b 00 00 00 18 00 00 00  |................|
-000007e0  39 45 f5 ea 0b 6e 3b 3b  ba 71 2b 6d e9 69 0a 0a  |9E...n;;.q+m.i..|
+000007e0  0a c9 64 b8 eb f2 39 97  b9 ea b9 30 3d a0 3f 2f  |..d...9....0=.?/|
 000007f0  32 00 00 00 20 00 00 00  01 00 00 00 00 00 0d 00  |2... ...........|
 00000800  00 01 0d 00 01 00 00 00  03 00 00 00 00 01 34 03  |..............4.|
 00000810  2a 00 00 00 10 00 00 00  00 00 00 00 00 00 00 00  |*...............|
@@ -74414,9 +74414,9 @@
 0012ff80  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 0012ffa0  00 06 80 00 00 00 00 00  00 00 00 00 61 2e 6f 75  |............a.ou|
-0012ffb0  74 00 21 40 04 9e 12 52  25 b7 3f 1f 1c 73 1f e2  |t.!@...R%.?..s..|
-0012ffc0  6a 66 ba 9e 1d 06 01 62  9c 52 53 97 e4 69 e2 16  |jf.....b.RS..i..|
-0012ffd0  76 c5 ad 7f ac b2 58 6f  c6 e9 66 c0 04 d7 d1 d1  |v.....Xo..f.....|
+0012ffb0  74 00 be f5 bf aa 8d 4d  87 8a b8 93 bf bd 90 fc  |t......M........|
+0012ffc0  fe 3e f6 e7 54 cb 57 58  fe 78 bd e5 f8 cf fa 57  |.>..T.WX.x.....W|
+0012ffd0  c9 87 ad 7f ac b2 58 6f  c6 e9 66 c0 04 d7 d1 d1  |......Xo..f.....|
 0012ffe0  6b 02 4f 58 05 ff 7c b4  7c 7a 85 da bd 8b 48 89  |k.OX..|.|z....H.|
 0012fff0  2c a7 ad 7f ac b2 58 6f  c6 e9 66 c0 04 d7 d1 d1  |,.....Xo..f.....|
 00130000  6b 02 4f 58 05 ff 7c b4  7c 7a 85 da bd 8b 48 89  |k.OX..|.|z....H.|
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Feb 16, 2023
@misha-ridge
Copy link
Author

Temporary directory is in LC_ID_DYLIB Mach-O command, so it looks like an oversight.

@misha-ridge misha-ridge changed the title plugin: macOS builds are not reproducible cmd/go: macOS builds are not reproducible Feb 16, 2023
@misha-ridge misha-ridge changed the title cmd/go: macOS builds are not reproducible cmd/go: macOS plugin builds are not reproducible Feb 16, 2023
@bcmills
Copy link
Contributor

bcmills commented Feb 16, 2023

It looks like there is an existing case here that attempts to mitigate LC_ID_DYLIB contamination, but it only currently applies for -buildmode=c-shared. Presumably it should be used for other build modes involving shared libraries?

https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/work/gc.go;l=672-684;drc=9565d990f4e2630dbf37c636299b9ddbd58b4f6b

@bcmills bcmills added this to the Go1.21 milestone Feb 16, 2023
@bcmills bcmills self-assigned this Feb 16, 2023
@bcmills bcmills added the NeedsFix The path to resolution is known, but the work has not been done. label Feb 16, 2023
@mknyszek mknyszek removed the compiler/runtime Issues related to the Go compiler and/or runtime. label Feb 24, 2023
@gopherbot
Copy link

Change https://go.dev/cl/477296 mentions this issue: cmd/go: make sure linker -o for plugin doesn't include tempdir path

gopherbot pushed a commit that referenced this issue Mar 17, 2023
@matloob
cmd/go: make sure linker -o for plugin doesn't include tempdir path
50479b7 
There is already a case that when buildmode=shared passes only the
basename of the -o argument to the link command to the linker (and
runs in the directory of that argument) to avoid having that
(temporary) directory of the file be included in the LC_ID_DYLIB load
command. Extend the case to buildmode=plugin, because the same thing
can happen there.

This can only happen on darwin: the -o command can be embedded into
Mach-O and PE binaries, but plugin isn't supported on Windows.

For #58557

Change-Id: I7a4a5627148e77c6906ac4583af3d9f053d5b249
Reviewed-on: https://go-review.googlesource.com/c/go/+/477296
Run-TryBot: Michael Matloob <matloob@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/478196 mentions this issue: cmd/go: extend the linker -o workaround for plugins to all platforms

gopherbot pushed a commit that referenced this issue Mar 21, 2023
@cherrymui
cmd/go: extend the linker -o workaround for plugins to all platforms
3aa7ada 
On Linux, for a shared object, at least with the Gold linker, the
output file path is recorded in the .gnu.version_d section. When
the output file path is in a temporary directory, it causes
nondeterministic build.

This is similar to #58557, but for Linux with the Gold linker.
Apply the same workaround as in CL 477296.

Should fix the linux-arm64-longtest builder.

Change-Id: Ic703bff32c1bcc40054b89be696e04280855e876
Reviewed-on: https://go-review.googlesource.com/c/go/+/478196
Run-TryBot: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
@matloob matloob closed this as completed Mar 30, 2023
@matloob matloob assigned matloob and unassigned bcmills Mar 30, 2023
@gopherbot
Copy link

Change https://go.dev/cl/494577 mentions this issue: cmd/go: diff .so files quietly in TestScript/build_plugin_reproducible

gopherbot pushed a commit that referenced this issue May 12, 2023
@dmitshur @gopherbot
cmd/go: diff .so files quietly in TestScript/build_plugin_reproducible
ed3ea52 
This avoids printing verbose binary data and making bell sounds when the
test fails. The binary data can be inspected via other means if needed.

For #58557.

Change-Id: Ia1c4f2c6b9ff2cf6f97611cf335b978fc7bb201f
Reviewed-on: https://go-review.googlesource.com/c/go/+/494577
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
@bcmills bcmills mentioned this issue Mar 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@matloob matloob

Labels
GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done. OS-Darwin
Projects
None yet
Milestone
Go1.21
Development

No branches or pull requests
5 participants
@mknyszek @bcmills @gopherbot @matloob @misha-ridge