Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/compile: builds not reproducible if input file order is different #38186

Closed
abhi opened this issue Mar 31, 2020 · 3 comments
Closed

cmd/compile: builds not reproducible if input file order is different #38186

abhi opened this issue Mar 31, 2020 · 3 comments
Labels
FrozenDueToAge

Comments

@abhi
Copy link

abhi commented Mar 31, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14.1 darwin/amd64

Does this issue reproduce with the latest release?

yes
Tried on go1.13.1 as well

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/aprativadi/Library/Caches/go-build"
GOENV="/Users/aprativadi/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/aprativadi/"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/hz/x4j3f51s6w1b1qhn82zcm__h3zmk4d/T/go-build924068317=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

We use a wrapper build tool buck that invokes the Go compile and link tools to build a go binary. We observed that go binaries are not reproducible if the ordering of files to the compile tool is not deterministic.

Create 2 files

$ cat cmd/dummy/main.go 
package main

import "fmt"

func main() {
  fmt.Println("dummy")
}

$cat cmd/dummy/build-info.go 
package main

func init(){
}

And run the compile step

$ go tool compile -p main -pack -nolocalimports -o dummy.a  -complete cmd/dummy/build-info.go cmd/dummy/main.go
$ md5 -r dummy.a 
2e02bda1e512c14c66a8937d91b4d4fe dummy.a

Change the order of files:

$ go tool compile -p main -pack -nolocalimports -o dummy.a  -complete cmd/dummy/main.go cmd/dummy/build-info.go
$ md5 -r dummy.a 
eebf4a84a56113ef049372ca6624cec1 dummy.a

What did you expect to see?

We expected to see the md5 hash of the .a object file to be the same

What did you see instead?

We observed the hashes are different if the ordering of files are different.

If this is not a compile tool bug, we could may be update the go compile tool doc ?
cc @codyohlsen
@ianlancetaylor
Copy link
Contributor

CC @randall77 @griesemer

At first glance I don't see any reason to expect the compiler to produce the same results regardless of input file order. It certainly essential for the compiler to produce the same results given the same arguments. But I'm not sure why it's important to keep the results the same if the input files are in a different order.

@griesemer
Copy link
Contributor

This is working as intended, at least as far as the compiler is concerned. Here are some relevant sections from the spec:

The declaration order of variables declared in multiple files is determined by the order in which the files are presented to the compiler: Variables declared in the first file are declared before any of the variables declared in the second file, and so on.

A package with no imports is initialized by assigning initial values to all its package-level variables followed by calling all init functions in the order they appear in the source, possibly in multiple files, as presented to the compiler.

To ensure reproducible initialization behavior, build systems are encouraged to present multiple files belonging to the same package in lexical file name order to a compiler.
(https://tip.golang.org/ref/spec#Package_initialization)

In short, the order in which files are presented to the compiler matters. It is the responsibility of the build tool to provide the files in some reproducible order.

I am going to close this since the spec is pretty clear. I believe the go tool sorts the filenames for reproducible behavior (cc: @bcmills for input).

@abhi
Copy link
Author

abhi commented Apr 1, 2020

@griesemer @ianlancetaylor thanks for prompt response and clarification. The only reason this example was interesting was due to init. We were under the impression that it would have an effect in ordering based on what you pointed out as well.

@golang golang locked and limited conversation to collaborators Apr 1, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@abhi @ianlancetaylor @griesemer @gopherbot