Comment 1:

Status changed to Accepted.

Sounds like this has been fixed. See #13247.

FWIW, I just checked on tip on a medium size project on darwin, and there are still two small differences:

--- ndsemu.0.hex    2016-05-05 21:10:35.000000000 +0200
+++ ndsemu.1.hex    2016-05-05 21:10:25.000000000 +0200
@@ -469452,7 +469452,7 @@
 0073bee0  00 a0 f6 b1 02 00 00 00  00 00 00 00 00 00 00 00  |................|
 0073bef0  02 00 00 00 64 00 00 00  00 00 00 00 00 00 00 00  |....d...........|
 0073bf00  26 00 00 00 64 00 00 00  00 00 00 00 00 00 00 00  |&...d...........|
-0073bf10  29 00 00 00 66 03 01 00  80 9a 2b 57 00 00 00 00  |)...f.....+W....|
+0073bf10  29 00 00 00 66 03 01 00  60 9a 2b 57 00 00 00 00  |)...f...`.+W....|
 0073bf20  71 00 00 00 26 03 00 00  10 39 00 04 00 00 00 00  |q...&....9......|
 0073bf30  7e 00 00 00 26 03 00 00  10 39 00 04 00 00 00 00  |~...&....9......|
 0073bf40  89 00 00 00 26 03 00 00  50 39 00 04 00 00 00 00  |....&...P9......|
@@ -493095,7 +493095,7 @@
 00798490  6c 64 65 72 73 2f 6c 77  2f 6a 64 62 6b 37 70 5f  |lders/lw/jdbk7p_|
 007984a0  64 34 67 6a 36 71 70 79  64 63 7a 70 62 77 32 30  |d4gj6qpydczpbw20|
 007984b0  38 30 30 30 30 67 6e 2f  54 2f 67 6f 2d 6c 69 6e  |80000gn/T/go-lin|
-007984c0  6b 2d 32 39 39 31 33 36  32 35 34 2f 67 6f 2e 6f  |k-299136254/go.o|
+007984c0  6b 2d 32 38 30 38 31 36  33 38 30 2f 67 6f 2e 6f  |k-280816380/go.o|
 007984d0  00 72 75 6e 74 69 6d 65  2e 74 65 78 74 00 67 6f  |.runtime.text.go|
 007984e0  2e 62 75 69 6c 64 69 64  00 6d 61 69 6e 2e 4e 65  |.buildid.main.Ne|
 007984f0  77 48 77 42 61 63 6b 75  70 52 61 6d 00 6d 61 69  |wHwBackupRam.mai|

The first one I guess it's a timestamp, the second seems the path to a temporary directory created by/for go-link.

Reopening. As far as I know the problem is Darwin-specific.

CC @neild

Sigh.

I guess it will keep regressing until some tests are added to trybots. It looks like the typical feature that is impossible not to break without tests.

There is a test, actually: TestCgoConsistentResults in src/cmd/go.

Works On My Machine(tm). Try running that test:

cd src/cmd/go
go test . -run=Consistent -v

Do you see -fdebug-prefix-map in the compiler invocation, or is the test skipped because the compiler does not support that flag?

If the test passes for you, can you give me steps to reproduce the inconsistency?

Thanks.

The test passes, and -fdebug-prefix-map is being passed.

To reproduce what I see, it should be sufficient something like this:

brew install sdl2
git clone https://github.com/rasky/ndsemu $GOPATH/src/ndsemu
cd $GOPATH/src/ndsemu
go get
go build ; md5sum ./ndsemu
go build ; md5sum ./ndsemu

@neild let me know if you reproduce and/or you need more info; i'd like to make sure that the bug is properly filed (though I don't personally need a quick resolution).

It looks like the problem here is that the external linker adds a symbol to the symbol table, containing the full path of the go.o file in the temporary directory.

I think we might need something similar to this fix:

but for macho instead of elf.

@ianlancetaylor @bradfitz Hi, by "varying the build path" I mean we are doing the first build in /build-1st and the second build in /build-2nd, and the resulting binaries contain differences.

Example diff and a list of things we vary.

I originally thought to open a new bug, because this bug seems to be about cgo and $WORK. We have also experienced this cgo issue elsewhere which we document here, but the issue I described with .gopclntab is different.

I could retitle the other bug to something a bit more specific, if you prefer. Such as "golang embeds build path into .gopclntab, causing loss of reproducibility".

Oh, I see. They are slightly different bugs. Okay, we can track them separately. I'll reopen #16860 and we can give it a better title.

@ianlancetaylor or @neild, didn't you figure out the magic flag

Confirmed this is still a problem:

$ go build -ldflags -linkmode=external -o gofmt1 cmd/gofmt
$ go build -ldflags -linkmode=external -o gofmt2 cmd/gofmt
$ cmp gofmt1 gofmt2
gofmt1 gofmt2 differ: char 3355697, line 14335
$ hexdump -C gofmt1 >gofmt1.hex
$ hexdump -C gofmt2 >gofmt2.hex
$ diff gofmt1.hex gofmt2.hex
208488c208488
< 00333430  d4 15 4c 59 00 00 00 00  58 03 00 00 26 03 00 00  |..LY....X...&...|
---
> 00333430  dc 15 4c 59 00 00 00 00  58 03 00 00 26 03 00 00  |..LY....X...&...|
218403,218404c218403,218404
< 00359fe0  2f 67 6f 2d 6c 69 6e 6b  2d 34 34 37 31 39 35 39  |/go-link-4471959|
< 00359ff0  38 34 2f 67 6f 2e 6f 00  72 75 6e 74 69 6d 65 2e  |84/go.o.runtime.|
---
> 00359fe0  2f 67 6f 2d 6c 69 6e 6b  2d 32 30 38 38 31 36 34  |/go-link-2088164|
> 00359ff0  38 36 2f 67 6f 2e 6f 00  72 75 6e 74 69 6d 65 2e  |86/go.o.runtime.|
$ 

I have no solution. I tried hacking cmd/link to pass a relative path (just "go.o") to the external linker and then running the linker in the temp directory (to make the relative path valid), hoping that the linker would then record the relative path instead of the absolute path seen in the diff. But that didn't help. (Also, in general I'm not sure of the implications of changing the directory where the linker runs, but it didn't help anyway.)

If someone knows how to tell clang on macOS not to record the absolute object paths used during a link, please let us know.

@rsc Here are a list of known issues with reproducible builds: https://tests.reproducible-builds.org/debian/index_issues.html perhaps this one has been studied and solved before for other packages.

Golang-1.7 is listed here: https://tests.reproducible-builds.org/debian/issues/unstable/captures_build_path_issue.html and discusses some of the issues.

HTH

@purpleidea Note that there are no known problems with reproducible builds with Go when using GCC, only when using clang.

@purpleidea Thanks. Assuming that is not on MacOS, please file a different issue with a reproduction case.

This is fixed as of now. There were a few different steps and steps backward.

@rsc Great, thanks! What does it mean: "There were a few different steps and steps backward." ?

Cheers!

It means there were multiple things to fix (steps forward) and maybe also a few new bugs introduced (steps backward) that also needed fixing.