Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: go buildid actionID part is not reproducible with clean cache #59553

Closed
smira opened this issue Apr 11, 2023 · 6 comments
Closed

cmd/go: go buildid actionID part is not reproducible with clean cache #59553

smira opened this issue Apr 11, 2023 · 6 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@smira
Copy link

smira commented Apr 11, 2023 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.20.3 linux/amd64

Does this issue reproduce with the latest release?

Yes, but it looks like it's a regression from go1.19.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/smira/.cache/go-build"
GOENV="/home/smira/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/smira/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/smira/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/smira/sdk/go1.20.3"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/smira/sdk/go1.20.3/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/smira/Documents/talos/go.mod"
GOWORK="/home/smira/Documents/talos/go.work"
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1337074187=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ git clone https://github.com/siderolabs/talos.git
$ cd talos/internal/app/init
$ go build -o /tmp/init1
$ go clean -cache
$ go build -o /tmp/init2
$ go tool buildid /tmp/init1
sKFzCzkNqNstkxeVQjx6/ur3V8IjUjAaSiSK8FRyf/GkeyU-h8TOuClgbmrAfr/wuYOkIb0eDRiRJoM6Gof
$ go tool buildid /tmp/init2
TzIKQctZITh9Dzu1A0R2/3Z-ERPrw_PhIW3bIu8R2/GkeyU-h8TOuClgbmrAfr/wuYOkIb0eDRiRJoM6Gof

What did you expect to see?

The buildid stays the same.

What did you see instead?

The buildid changes every time I run a build with a clean cache.

The contentID part stays the same, but actionID is always different.

I can't reproduce it with other Go programs, but Talos source code exhibits this issue.

If I repeat the same steps e.g. with go1.19.4, the buildid stays the same.

I noticed this issue initially with our Dockerized build environment, but I reduced it down to a single binary.

If inspected with diffoscope, the only actual difference in the binary is the buildid itself:

├── readelf --wide --notes {}
│ @@ -1,4 +1,4 @@
│  
│  Displaying notes found in: .note.go.buildid
│    Owner                Data size 	Description
│ -  Go                   0x00000053	GO BUILDID	   description data: 73 4b 46 7a 43 7a 6b 4e 71 4e 73 74 6b 78 65 56 51 6a 78 36 2f 75 72 33 56 38 49 6a 55 6a 41 61 53 69 53 4b 38 46 52 79 66 2f 47 6b 65 79 55 2d 68 38 54 4f 75 43 6c 67 62 6d 72 41 66 72 2f 77 75 59 4f 6b 49 62 30 65 44 52 69 52 4a 6f 4d 36 47 6f 66 
│ +  Go                   0x00000053	GO BUILDID	   description data: 54 7a 49 4b 51 63 74 5a 49 54 68 39 44 7a 75 31 41 30 52 32 2f 33 5a 2d 45 52 50 72 77 5f 50 68 49 57 33 62 49 75 38 52 32 2f 47 6b 65 79 55 2d 68 38 54 4f 75 43 6c 67 62 6d 72 41 66 72 2f 77 75 59 4f 6b 49 62 30 65 44 52 69 52 4a 6f 4d 36 47 6f 66 
├── strings --all --bytes=8 {}
│ @@ -1,8 +1,8 @@
│ -sKFzCzkNqNstkxeVQjx6/ur3V8IjUjAaSiSK8FRyf/GkeyU-h8TOuClgbmrAfr/wuYOkIb0eDRiRJoM6Gof
│ +TzIKQctZITh9Dzu1A0R2/3Z-ERPrw_PhIW3bIu8R2/GkeyU-h8TOuClgbmrAfr/wuYOkIb0eDRiRJoM6Gof
│  /lib64/ld-linux-x86-64.so.2
│  []A\A]A^A_
│  [A\A]A^]
│  []A\A]A^A_
│  SUATAUAVAWH
│  A_A^A]A\][
@smira smira changed the title affected/package: go buildid actionID part is not reproducible with clean cache Apr 11, 2023
@smira smira changed the title go buildid actionID part is not reproducible with clean cache cmd/compile: go buildid actionID part is not reproducible with clean cache Apr 11, 2023
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Apr 11, 2023
@smira smira changed the title cmd/compile: go buildid actionID part is not reproducible with clean cache cmd/go: go buildid actionID part is not reproducible with clean cache Apr 11, 2023
@smira
Copy link
Author

smira commented Apr 11, 2023 •
edited

A few more notes:

  1. Not cleaning the cache between the rebuilds produces identical buildid.
  2. Adding -ldflags "buildid=" provides reproducible buildid. (wrong test, it's different each time still)
  3. Adding -ldflags all=-buildid= removes buildid and produces identical binary, like in cmd/go: executables built with -trimpath are not reproducible with development Go toolchains unless -ldflags removes -buildid= #59525

smira added a commit to smira/talos that referenced this issue Apr 12, 2023
@smira
chore: remove Go buildid
bd1cff3 
See golang/go#59553

The problem is that `buildid` is not reproducible for Talos builds with
Go 1.20+. Remove the `buildid` until the issue is resolved to make
builds reproducible.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
@smira smira mentioned this issue Apr 12, 2023
Merged
@bcmills
Copy link
Contributor

bcmills commented Apr 12, 2023

Could be #59571, perhaps?

Or, since you are building with CGO_ENABLED, perhaps your C toolchain does not support reproducible builds (compare #56896). What C compiler are you using?

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 12, 2023
@bcmills bcmills added this to the Backlog milestone Apr 12, 2023
@bcmills bcmills added GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Apr 12, 2023
@smira
Copy link
Author

smira commented Apr 12, 2023

Could be #59571, perhaps?

yes, we do use generics, but go1.19 doesn't show this behavior.

What is the best way for me to check for this? I'm not sure how can I easily see what goes into actionID so I can diff between the builds.

Or, since you are building with CGO_ENABLED, perhaps your C toolchain does not support reproducible builds (compare #56896). What C compiler are you using?

For this case I was building on latest Ubuntu, so pretty much gcc which ships with it.

But I tried with CGO_ENABLED=0 as well, and it behaves same way. The issue started for me in our hermetic build environment which is completely reproducible, but I reduced it back to regular go build invocation.

@rsc
Copy link
Contributor

rsc commented Apr 12, 2023

This is almost certainly #59571. Go 1.19 didn't have the bug. Go 1.20 does. The easiest thing to do is wait for #59571 to be fixed and then see if you still have the problem.

@rsc
Copy link
Contributor

rsc commented Apr 13, 2023

@smira if you build Go from the current development branch I believe you'll see the problem is fixed.

@smira
Copy link
Author

smira commented Apr 13, 2023

@rcs thank you, verfied with gotip, the problem is fixed.

We'll be waiting for go1.20.4!

@smira smira closed this as completed Apr 13, 2023
smira added a commit to smira/talos that referenced this issue Apr 18, 2023
@smira
chore: remove Go buildid
8b22e7e 
See golang/go#59553

The problem is that `buildid` is not reproducible for Talos builds with
Go 1.20+. Remove the `buildid` until the issue is resolved to make
builds reproducible.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
(cherry picked from commit bd1cff3)
@golang golang locked and limited conversation to collaborators Apr 12, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@smira @rsc @bcmills @gopherbot