Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go, cmd/cgo: repeatable builds with Clang 3.8 #15405

Closed
EtienneBruines opened this issue Apr 21, 2016 · 7 comments
Closed

cmd/go, cmd/cgo: repeatable builds with Clang 3.8 #15405

EtienneBruines opened this issue Apr 21, 2016 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Go1.9

Comments

@EtienneBruines
Copy link

EtienneBruines commented Apr 21, 2016 •
edited

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
    go version devel +f8fc371 Thu Apr 21 17:04:01 2016 +0000 linux/amd64
    (but GOROOT_BOOTSTRAP is set to an installation of go1.6.2)
  2. What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/etiennebruines/workspaces/go"
GORACE=""
GOROOT="/home/etiennebruines/workspaces/go-master"
GOTOOLDIR="/home/etiennebruines/workspaces/go-master/pkg/tool/linux_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build179019630=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

(Tested different versions of `clang` (with different results!), more info down below. )
  1. What did you do?
    Attempted to build master branch of Go (currently HEAD is at 40f1d0c )
    (more specifically: ./all.bash within the src directory)
  2. What did you expect to see?
    No errors.
  3. What did you see instead?
    An error within the tests, namely:
--- FAIL: TestCgoConsistentResults (1.10s)
    go_test.go:244: running testgo [build -o /tmp/gotest260338695/cgotest1 cgotest]
    go_test.go:244: running testgo [build -x -o /tmp/gotest260338695/cgotest2 cgotest]
    go_test.go:263: standard error:
    go_test.go:264: WORK=/tmp/go-build737947577
        mkdir -p $WORK/cgotest/_obj/
        mkdir -p $WORK/
        cd /home/etiennebruines/workspaces/go-master/src/cmd/go/testdata/src/cgotest
        CGO_LDFLAGS="-g" "-O2" /home/etiennebruines/workspaces/go-master/pkg/tool/linux_amd64/cgo -objdir $WORK/cgotest/_obj/ -importpath cgotest -- -I $WORK/cgotest/_obj/ m.go
        cd $WORK
        clang-3.8 -fdebug-prefix-map=a=b -c trivial.c
        clang-3.8 -gno-record-gcc-switches -c trivial.c
        cd /home/etiennebruines/workspaces/go-master/src/cmd/go/testdata/src/cgotest
        clang-3.8 -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I $WORK/cgotest/_obj/ -g -O2 -o $WORK/cgotest/_obj/_cgo_main.o -c $WORK/cgotest/_obj/_cgo_main.c
        clang-3.8 -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I $WORK/cgotest/_obj/ -g -O2 -o $WORK/cgotest/_obj/_cgo_export.o -c $WORK/cgotest/_obj/_cgo_export.c
        clang-3.8 -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I $WORK/cgotest/_obj/ -g -O2 -o $WORK/cgotest/_obj/m.cgo2.o -c $WORK/cgotest/_obj/m.cgo2.c
        clang-3.8 -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -o $WORK/cgotest/_obj/_cgo_.o $WORK/cgotest/_obj/_cgo_main.o $WORK/cgotest/_obj/_cgo_export.o $WORK/cgotest/_obj/m.cgo2.o -g -O2
        /home/etiennebruines/workspaces/go-master/pkg/tool/linux_amd64/cgo -objdir $WORK/cgotest/_obj/ -dynpackage cgotest -dynimport $WORK/cgotest/_obj/_cgo_.o -dynout $WORK/cgotest/_obj/_cgo_import.go
        cd $WORK
        clang-3.8 -no-pie -c trivial.c
        cd /home/etiennebruines/workspaces/go-master/src/cmd/go/testdata/src/cgotest
        clang-3.8 -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -o $WORK/cgotest/_obj/_all.o $WORK/cgotest/_obj/_cgo_export.o $WORK/cgotest/_obj/m.cgo2.o -g -O2 -Wl,-r -nostdlib -Wl,--build-id=none
        /home/etiennebruines/workspaces/go-master/pkg/tool/linux_amd64/compile -o $WORK/cgotest.a -trimpath $WORK -p cgotest -buildid 1491fc40a2d4f2376923084c60120b2fd293fe0e -D _/home/etiennebruines/workspaces/go-master/src/cmd/go/testdata/src/cgotest -I $WORK -pack $WORK/cgotest/_obj/_cgo_gotypes.go $WORK/cgotest/_obj/m.cgo1.go $WORK/cgotest/_obj/_cgo_import.go
        pack r $WORK/cgotest.a $WORK/cgotest/_obj/_all.o # internal
        mkdir -p /tmp/gotest260338695/
        mv $WORK/cgotest.a /tmp/gotest260338695/cgotest2

    go_test.go:2775: building cgotest twice did not produce the same output
FAIL
FAIL    cmd/go  29.068s

Notes

When running CGO_ENABLED=0, all errors go away (makes sense).
@ianlancetaylor ianlancetaylor changed the title TestCgoConsistentResults cmd/go: TestCgoConsistentResults fails with clang 3.8 Apr 21, 2016
@ianlancetaylor ianlancetaylor added this to the Go1.7 milestone Apr 21, 2016
@bradfitz bradfitz mentioned this issue May 17, 2016
Closed
@wheelcomplex wheelcomplex mentioned this issue May 20, 2016
Closed
@quentinmit quentinmit added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 26, 2016
@rsc
Copy link
Contributor

rsc commented May 27, 2016

This is unfortunate but not critical for the release.

@rsc rsc modified the milestones: Go1.7Maybe, Go1.7 May 27, 2016
@mikioh mikioh changed the title cmd/go: TestCgoConsistentResults fails with clang 3.8 cmd/go: TestCgoConsistentResults fails with clang 3.8 or gcc 4.3 Jun 3, 2016
@ianlancetaylor
Copy link
Contributor

Postponing to 1.8.

@ianlancetaylor ianlancetaylor modified the milestones: Go1.8, Go1.7Maybe Jun 9, 2016
@mewmew mewmew mentioned this issue Aug 8, 2016
Closed
@rsc
Copy link
Contributor

rsc commented Oct 21, 2016

It sounds like we just need to wait for clang and gcc to fix their reproducible build support.

clang

Support for -fdebug-prefix-map was added Oct 2015. Clang 3.7.0 was released Sep 2015, Clang 3.7.1 was released Jan 2016, and Clang 3.8 was released Mar 2016. My guess is the change may have missed the Clang 3.7 point release but was likely in Clang 3.8. Without detecting support for the flag, the test is skipped. This is why we don't see failures before Clang 3.8.

The linux-amd64-clang gomote image is using Clang 3.5, so we don't test this case on the builder.

@bradfitz, is it possible to update to a newer clang? Latest is 3.9.0 released Sep 2 2016. http://llvm.org/releases/

gcc

Support for -fdebug-prefix-map is more widespread, but some versions don't quite work. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69821 shows that while -fdebug-prefix-map did the right thing to the debug symbols, gcc then included that option in the DW_AT_producer field of the dwarf. If the goal of -fdebug-prefix-map=old=new is to eliminate the string "old" from the binary, writing down that option into the binary defeats the purpose. The bug entry says the target milestone is gcc 6 and the discussion there sounds like the decision was not to backport to gcc 5. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=819176, the Debian entry for this bug, suggests it is fixed in Debian's gcc 5.4.0. It is unclear whether that means the fix is only in Debian's copy or is upstream. It's unclear to me when DW_AT_producer first started recording the whole command line.

Obviously the tests do pass on our gcc builders, and I confirmed that (unlike the clang builder) the gcc on the builder does support the flag, so the test should be running. Perhaps gcc 4.3 had a related problem.

@rsc rsc unassigned neild Oct 21, 2016
@rsc rsc modified the milestones: Go1.9, Go1.8 Oct 21, 2016
@rsc
Copy link
Contributor

rsc commented Oct 21, 2016

See also #13247 and #9206.

@rsc rsc changed the title cmd/go: TestCgoConsistentResults fails with clang 3.8 or gcc 4.3 cmd/go, cmd/cgo: repeatable builds with Clang 3.8 Oct 21, 2016
@ALTree
Copy link
Member

ALTree commented Oct 27, 2016

Also #5170.

@ianlancetaylor ianlancetaylor mentioned this issue Dec 16, 2016
Closed
@gopherbot
Copy link

CL https://golang.org/cl/34675 mentions this issue.

gopherbot pushed a commit that referenced this issue Dec 22, 2016
@cixtor
cmd/go: skip flaky TestCgoConsistentResults on FreeBSD
1719034 
FreeBSD 11 or above uses clang-3.6 or higher by default.

Updates #15405.

Change-Id: If49ce298130165f9e1525c7fd0fd5aa39099ad53
Reviewed-on: https://go-review.googlesource.com/34675
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@rsc
Copy link
Contributor

rsc commented Jun 22, 2017

This is working for me:

$ clang --version
clang version 3.8.0-2ubuntu3~trusty4 (tags/RELEASE_380/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
$ CC=clang go test -run=CgoConsistent -v
=== RUN   TestCgoConsistentResults
--- PASS: TestCgoConsistentResults (0.94s)
	go_test.go:310: running testgo [build -o /tmp/gotest742500210/cgotest1 cgotest]
	go_test.go:310: running testgo [build -x -o /tmp/gotest742500210/cgotest2 cgotest]
	go_test.go:323: standard error:
	go_test.go:324: WORK=/tmp/go-build667043988
		mkdir -p $WORK/cgotest/_obj/
		mkdir -p $WORK/
		cd /home/rsc/go/src/cmd/go/testdata/src/cgotest
		CGO_LDFLAGS="-g" "-O2" /home/rsc/go/pkg/tool/linux_amd64/cgo -objdir $WORK/cgotest/_obj/ -importpath cgotest -- -I $WORK/cgotest/_obj/ -g -O2 m.go
		cd $WORK
		clang -fdebug-prefix-map=a=b -c trivial.c
		clang -gno-record-gcc-switches -c trivial.c
		cd $WORK/cgotest/_obj
		clang -I /home/rsc/go/src/cmd/go/testdata/src/cgotest -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_cgo_export.o -c _cgo_export.c
		clang -I /home/rsc/go/src/cmd/go/testdata/src/cgotest -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./m.cgo2.o -c m.cgo2.c
		clang -I /home/rsc/go/src/cmd/go/testdata/src/cgotest -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_cgo_main.o -c _cgo_main.c
		cd /home/rsc/go/src/cmd/go/testdata/src/cgotest
		clang -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -o $WORK/cgotest/_obj/_cgo_.o $WORK/cgotest/_obj/_cgo_main.o $WORK/cgotest/_obj/_cgo_export.o $WORK/cgotest/_obj/m.cgo2.o -g -O2
		/home/rsc/go/pkg/tool/linux_amd64/cgo -dynpackage cgotest -dynimport $WORK/cgotest/_obj/_cgo_.o -dynout $WORK/cgotest/_obj/_cgo_import.go
		cd $WORK
		clang -no-pie -c trivial.c
		cd /home/rsc/go/src/cmd/go/testdata/src/cgotest
		clang -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -o $WORK/cgotest/_obj/_all.o $WORK/cgotest/_obj/_cgo_export.o $WORK/cgotest/_obj/m.cgo2.o -g -O2 -Wl,-r -nostdlib -Wl,--build-id=none
		/home/rsc/go/pkg/tool/linux_amd64/compile -o $WORK/cgotest.a -trimpath $WORK -p cgotest -buildid 7216745655d9549736f59595dc578c7f54579d74 -D _/home/rsc/go/src/cmd/go/testdata/src/cgotest -I $WORK -pack -c=4 $WORK/cgotest/_obj/_cgo_gotypes.go $WORK/cgotest/_obj/m.cgo1.go $WORK/cgotest/_obj/_cgo_import.go
		pack r $WORK/cgotest.a $WORK/cgotest/_obj/_all.o # internal
		mkdir -p /tmp/gotest742500210/
		mv $WORK/cgotest.a /tmp/gotest742500210/cgotest2
		
PASS
ok  	cmd/go	2.705s
$

The builders are on clang 3.9 and it's working there too. Going to assume this is now fixed and that the problem before was that either the specific clang 3.8 being used didn't have -fdebug-prefix-map or did not implement it correctly.

@rsc rsc closed this as completed Jun 22, 2017
@golang golang locked and limited conversation to collaborators Jun 22, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Go1.9
Development

No branches or pull requests
7 participants
@neild @rsc @quentinmit @ianlancetaylor @ALTree @EtienneBruines @gopherbot