cc @cherrymui @rsc @ianlancetaylor

Thanks for the issue. Could you share how you invoke the build commands? Do you run go link (by which I think you mean go tool link) directly? Do you use go build -buildmode c-shared?

Hi! Yes to both of those things (we use the Buck build system which breaks down the process into individual invocations of Go tools like compile, link, asm or pack directly). We supply -buildmode c-shared to link.

I'm currently working on a minimalized repro case that only relies on pure Golang toolchain, but other things will keep me busy till about mid-July, so cannot provide more info at the moment.

What flags do you pass to compile and asm? Could you show a full list of commands you run? Thanks.

Note that not all compiler/assembler flags are compatible with the build mode.

The original build is quite extensive and complicated, but here's the gist of what's being run to produce some of the key .o's that feed into the problematic link:

compile -p main -pack -trimpath <ROOT_OF_SCM_REPO> -nolocalimports -dynlink '-buildid=' -o <OUTPUT_ARCHIVE>.a -I <INCLUDE_DIR> -complete <ADDITIONAL_BUILD_INFO_SYMBOLS>.go /PATH/TO/main.go

The asm invocation is only used on some of the Golang core crypto lib internals way earlier in the build dependency tree, not on any of our own sources:

asm -trimpath <ROOT_OF_SCM_REPO> -D GOOS_linux -D GOARCH_amd64 -o <OUTPUT_OBJECT>.o -I <INCLUDE_DIR> -I <golang.org/x/crypto/curve25519/internal/field/field_HEADERS_DIR> -I <golang.org/x/crypto/curve25519/internal/field/field_HEADERS_DIR> <GOLANG_SOURCE_TREE>/go/golang.org/x/crypto/curve25519/internal/field/fe_amd64.s

I believe this part is a recreation of Golang's own distribution build within Buck.

Is there perhaps some key arg to go compile that we aren't supplying that causes its produced .o to trigger go link switch to logic that chooses the LocalExec Thread Local Storage access model?

I think you need to pass the -shared flag to the compiler and the assembler. I don't think you want to pass the -dynlink flag.

Note that this also applies to the Go standard library packages. If you use pre-built packages for the standard library, make sure they are built with those flags as well.

Thanks for the suggestion! I'll try that and report back. It might need to wait till August, though, due to other things I need to work on first.

@cherrymui I created a minimal repro script.

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/maxshegai/.cache/go-build"
GOENV="/home/maxshegai/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/maxshegai/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/maxshegai/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3023500340=/tmp/go-build -gno-record-gcc-switches"

$ go version
go version go1.19 linux/amd64

$ clang++ -v
clang version 14.0.0 (Fedora 14.0.0-1.fc36)
Target: x86_64-redhat-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/12
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/12
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64

$ ld.lld -v
LLD 14.0.0 (compatible with GNU linkers)

Script

#!/bin/bash

AFILE="${PWD}"/repro.a
SOFILE="${PWD}"/repro.so

go tool compile -o "${AFILE}" "${PWD}"/main.go

go tool link -buildmode c-shared -linkmode external -extld /usr/bin/clang++ \
   "-extldflags=-fuse-ld=lld -o ${SOFILE}" \
   "${AFILE}"
EOF

$ cat main.go
package main

func main() {}

Output

$ ./repro.sh
/usr/local/go/pkg/tool/linux_amd64/link: running /usr/bin/clang++ failed: exit status 1
ld.lld: error: relocation R_X86_64_TPOFF32 against runtime.tlsg cannot be used with -shared
>>> defined in /tmp/go-link-202360648/go.o
>>> referenced by cpu_x86.go:57 (/usr/local/go/src/internal/cpu/cpu_x86.go:57)
>>>               /tmp/go-link-202360648/go.o:(internal/cpu.doinit)

ld.lld: error: relocation R_X86_64_TPOFF32 against runtime.tlsg cannot be used with -shared
>>> defined in /tmp/go-link-202360648/go.o
>>> referenced by cpu_x86.go:79 (/usr/local/go/src/internal/cpu/cpu_x86.go:79)
>>>               /tmp/go-link-202360648/go.o:(internal/cpu.doinit)

ld.lld: error: relocation R_X86_64_TPOFF32 against runtime.tlsg cannot be used with -shared
>>> defined in /tmp/go-link-202360648/go.o
>>> referenced by cpu_x86.go:85 (/usr/local/go/src/internal/cpu/cpu_x86.go:85)
>>>               /tmp/go-link-202360648/go.o:(internal/cpu.doinit)

...

# Similar errors referencing different go files

@n-canter , as commented in #53214 (comment) , you'd need to use the -shared flag for the compiler and assembler, including building the Go standard library. In general please use the go build command, instead of doing go tool compile and go tool link directly. Thanks.

Hey, @cherrymui, thanks for looking into this. We're using the lower level go tool compile/link as we've got to stay integrated with the cross-language build system deployed internally here at Meta.

We have our reservations about rebuilding the world with -shared. Can you briefly explain the possible implications of doing so, please?

In short -shared tells the compiler and assembler to build PIC code, so it can be used for c-shared, PIE, or plugin build mode.

I'd still suggest to use the go command if possible. Directly using the compiler and linker is not a supported mode and may change any time, especially with non-trivial flags such as c-shared build mode. If you really want to do that, maybe you could take a look at the compiler/assembler/linker invocations from the go command by doing go build -x or go build -n, and pass the same flags.

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

My last 2 cents after closure of this task in the interest of future follow-up discussions that may happen.

First of all, thanks @cherrymui for pointing out the necessity to rebuild all the dependencies with -shared! This allowed us to pinpoint the issue and work around it.

I can see that the world needs rebuilding with -shared, as the resultant final binary is compiled into machine code which needs to work somehow and meet the requirements of a DSO. There can't be any reservations about that.

However, I believe that this discussion brought to light another interesting fact, which to the best of my knowledge (and Googling) is not very well documented.

In our testing and repro work, we have noticed that the go build subcommand, when used in c-shared buildmode, has an implicit special case behavior relating to the standard library.

Namely, it spawns an implicit build job to perform temporary compilation of Golang standard library modules with the -shared flag, making use of an implied, universal availability of their sources.

Here's an easy way to observe this behavior:

$ cat go_build.sh
#!/bin/bash
go build -a -n -buildmode=c-shared -ldflags "-extld=/usr/bin/clang++ -extldflags=-fuse-ld=lld -o main.so" main.go 2>&1 | grep 'compile.*internal'

$ cat main.go
package main

func main() {}

$ ./go_build.sh
/usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/b004/_pkg_.a -trimpath "$WORK/b004=>" -p internal/goarch -std -+ -complete -installsuffix shared -buildid ZHkSq2swJTRI2k6gm2D5/ZHkSq2swJTRI2k6gm2D5 -goversion go1.18.4 -shared -c=4 -nolocalimports -importcfg $WORK/b004/importcfg -pack /usr/lib/golang/src/internal/goarch/goarch.go /usr/lib/golang/src/internal/goarch/goarch_amd64.go /usr/lib/golang/src/internal/goarch/zgoarch_amd64.go
/usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/b003/_pkg_.a -trimpath "$WORK/b003=>" -p internal/abi -std -+ -installsuffix shared -buildid w_fa4cF9I81-PRD0VKU4/w_fa4cF9I81-PRD0VKU4 -goversion go1.18.4 -symabis $WORK/b003/symabis -shared -c=4 -nolocalimports -importcfg $WORK/b003/importcfg -pack -asmhdr $WORK/b003/go_asm.h /usr/lib/golang/src/internal/abi/abi.go /usr/lib/golang/src/internal/abi/abi_amd64.go
/usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/b007/_pkg_.a -trimpath "$WORK/b007=>" -p internal/cpu -std -+ -installsuffix shared -buildid mBY4T5UiI7pryaN5qOLs/mBY4T5UiI7pryaN5qOLs -goversion go1.18.4 -symabis $WORK/b007/symabis -shared -c=4 -nolocalimports -importcfg $WORK/b007/importcfg -pack -asmhdr $WORK/b007/go_asm.h /usr/lib/golang/src/internal/cpu/cpu.go /usr/lib/golang/src/internal/cpu/cpu_x86.go
/usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/b006/_pkg_.a -trimpath "$WORK/b006=>" -p internal/bytealg -std -+ -installsuffix shared -buildid iBmsnO3jk4HE__XY62V5/iBmsnO3jk4HE__XY62V5 -goversion go1.18.4 -symabis $WORK/b006/symabis -shared -c=4 -nolocalimports -importcfg $WORK/b006/importcfg -pack -asmhdr $WORK/b006/go_asm.h /usr/lib/golang/src/internal/bytealg/bytealg.go /usr/lib/golang/src/internal/bytealg/compare_native.go /usr/lib/golang/src/internal/bytealg/count_native.go /usr/lib/golang/src/internal/bytealg/equal_generic.go /usr/lib/golang/src/internal/bytealg/equal_native.go /usr/lib/golang/src/internal/bytealg/index_amd64.go /usr/lib/golang/src/internal/bytealg/index_native.go /usr/lib/golang/src/internal/bytealg/indexbyte_native.go
/usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/b008/_pkg_.a -trimpath "$WORK/b008=>" -p internal/goexperiment -std -complete -installsuffix shared -buildid -JUqi24MGyMHg9nBuQLq/-JUqi24MGyMHg9nBuQLq -goversion go1.18.4 -shared -c=4 -nolocalimports -importcfg $WORK/b008/importcfg -pack /usr/lib/golang/src/internal/goexperiment/exp_fieldtrack_off.go /usr/lib/golang/src/internal/goexperiment/exp_heapminimum512kib_off.go /usr/lib/golang/src/internal/goexperiment/exp_pacerredesign_on.go /usr/lib/golang/src/internal/goexperiment/exp_preemptibleloops_off.go /usr/lib/golang/src/internal/goexperiment/exp_regabiargs_on.go /usr/lib/golang/src/internal/goexperiment/exp_regabireflect_on.go /usr/lib/golang/src/internal/goexperiment/exp_regabiwrappers_on.go /usr/lib/golang/src/internal/goexperiment/exp_staticlockranking_off.go /usr/lib/golang/src/internal/goexperiment/exp_unified_off.go /usr/lib/golang/src/internal/goexperiment/flags.go
/usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/b009/_pkg_.a -trimpath "$WORK/b009=>" -p internal/goos -std -+ -complete -installsuffix shared -buildid nAczB-jLkeyDmoPbSPMJ/nAczB-jLkeyDmoPbSPMJ -goversion go1.18.4 -shared -c=4 -nolocalimports -importcfg $WORK/b009/importcfg -pack /usr/lib/golang/src/internal/goos/goos.go /usr/lib/golang/src/internal/goos/zgoos_linux.go
/usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/b010/_pkg_.a -trimpath "$WORK/b010=>" -p runtime/internal/atomic -std -+ -installsuffix shared -buildid cqPR-KtXpcMsgrvGIPEQ/cqPR-KtXpcMsgrvGIPEQ -goversion go1.18.4 -symabis $WORK/b010/symabis -shared -c=4 -nolocalimports -importcfg $WORK/b010/importcfg -pack -asmhdr $WORK/b010/go_asm.h /usr/lib/golang/src/runtime/internal/atomic/atomic_amd64.go /usr/lib/golang/src/runtime/internal/atomic/doc.go /usr/lib/golang/src/runtime/internal/atomic/stubs.go /usr/lib/golang/src/runtime/internal/atomic/types.go /usr/lib/golang/src/runtime/internal/atomic/types_64bit.go /usr/lib/golang/src/runtime/internal/atomic/unaligned.go
/usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/b011/_pkg_.a -trimpath "$WORK/b011=>" -p runtime/internal/math -std -+ -complete -installsuffix shared -buildid zM2rfYoL8pHhlQ9JCx1h/zM2rfYoL8pHhlQ9JCx1h -goversion go1.18.4 -shared -c=4 -nolocalimports -importcfg $WORK/b011/importcfg -pack /usr/lib/golang/src/runtime/internal/math/math.go
/usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/b012/_pkg_.a -trimpath "$WORK/b012=>" -p runtime/internal/sys -std -+ -complete -installsuffix shared -buildid DP45ebbCVyK97YOvB_62/DP45ebbCVyK97YOvB_62 -goversion go1.18.4 -shared -c=4 -nolocalimports -importcfg $WORK/b012/importcfg -pack /usr/lib/golang/src/runtime/internal/sys/consts.go /usr/lib/golang/src/runtime/internal/sys/intrinsics.go /usr/lib/golang/src/runtime/internal/sys/intrinsics_common.go /usr/lib/golang/src/runtime/internal/sys/sys.go /usr/lib/golang/src/runtime/internal/sys/zversion.go
/usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/b013/_pkg_.a -trimpath "$WORK/b013=>" -p runtime/internal/syscall -std -installsuffix shared -buildid Zk3yU1uXw4CdQuVMDyE1/Zk3yU1uXw4CdQuVMDyE1 -goversion go1.18.4 -symabis $WORK/b013/symabis -shared -c=4 -nolocalimports -importcfg $WORK/b013/importcfg -pack -asmhdr $WORK/b013/go_asm.h /usr/lib/golang/src/runtime/internal/syscall/syscall_linux.go
/usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/b016/_pkg_.a -trimpath "$WORK/b016=>" -p internal/race -std -complete -installsuffix shared -buildid TkqfhZvzcqMe2AZz5SMH/TkqfhZvzcqMe2AZz5SMH -goversion go1.18.4 -shared -c=4 -nolocalimports -importcfg $WORK/b016/importcfg -pack /usr/lib/golang/src/internal/race/doc.go /usr/lib/golang/src/internal/race/norace.go
$

As one can see, without any explicit declarations, module sources like golang/src/runtime/internal/sys/sys.go get implicitly passed through go compile -shared with output sent to a temporary binary archive like $WORK/b012/_pkg_.a.

I'd like to argue that this is a huge and surprising departure from established practice with compiler and build toolchains in other compiled languages.

Typically, when the language's standard library needs special mode of compilation for a flavor of its binaries (be it target CPU architecture or shared vs static), it's done in advance and the compiled binaries get shipped along with "normal" ones. E.g. regular C libraries which usually ship both static archives and .so shared objects in separate precompiled files.

This makes many beneficial things easier, e.g. having the trusted compiled libraries authenticated and cryptographically signed; saving time on compilation etc. At the very least, this has the benefit of adhering to the principle of least astonishment for the user.

And Golang even has a facility for shipping different flavors of compiled standard libs, after all we have architecture-oriented dirs like go/pkg/darwin_amd64/, go/pkg/darwin_arm64/, go/pkg/linux_amd64/.

It is obviously up to the Golang project to decide how the build toolchain works, but I'd like to suggest changing the behavior from on-demand recompilation of standard library modules to shipping them pre-compiled in shared mode (e.g. in directories named like go/pkg/linux_amd64_shared/).

@aadamowski the build of the packages are cached, so it shouldn't be rebuilt the next time you build it with the same flag.

However, you used the -a flag at go build, which is specifically to force rebuilding everything. You can see that even without the special build mode a plain go build -a will rebuild all dependent packages.

I understand. However, performance is only a minor of all the concerns that I have around this design. It also applies only to local builds (on a specific machine/OS instance/filesystem), not any distributed build infrastructures (out of the box).

My largest concern is the departure from the established practices around how build toolchains generally work and what features their individual tools provide (e.g. a compiler does not suddenly decide on its own to compile a piece of source it wasn't explicitly asked to compile).

Sorry, I'm not sure I understand. You used the -a flag, which is explicitly asking to compile everything.

Oh, I see what you mean. Yes, I'd like to be able to drop the usage of -a from go build and still be able to produce a c-shared library with pre-compiled shared stdlib packages, and without a need to recompile and overwrite the deps when switching between c-shared and regular buildmodes (they'd need to be cached separately from each other).
In essence, I'd argue for go build to maintain separate caches for built artifacts, keyed on build architecture and shared vs regular (non-PIC) build modes.

IOW, the idea is to add another sublevel to module cache directory structure (https://go.dev/ref/mod#module-cache) to guarantee separation between compiled artifacts that cannot possibly work together (due to being cross-compiled for a different architecture, or mixing of -shared and non-shared build modes).

I'd argue for go build to maintain separate caches for built artifacts, keyed on build architecture and shared vs regular (non-PIC) build modes.

It does already, as well as on other compile/linker flags. The use of -a shouldn't be necessary (for building using the go command; if you want to let it print the subcommand invocations and run them manually, -a may still be a good idea for completeness).

I see. Thanks, I think we now have a better understanding of it now and how to adapt Buck's Golang support to handle c-shared gracefully and in a way that aligns with go build's philosophy.

Would you have some time to answer the last few questions?

AFAIU the behavior you mentioned for keying on compile/linker flags only applies to the build cache, correct? E.g. in my case, after dropping -a I've observed go build to use a stable, deterministic path of .cache/go-build/4c/4c90960653622732a99fe2bbbc13bb81c3f3f3c66a1c5c5f44a36f493f0130be-d. I assume the last part is a hash of the relevant compilation/build flags derived from the Action ID? For deeper study, do I understand correctly that the code responsible for this is go/src/cmd/go/internal/work/buildid.go? I was trying to locate some documentation around the Action ID design but only found source code so far.

Is there any similar mechanism that would apply to pre-compiled modules originating externally (installed on the system, not produced by the local build environment) and where the go tools search for them?
Such mechanism, AFAIU, would need to be more coarse than Action ID which seems to be very fine-grained (one of the inputs is the hash of all the combined build dependencies) and doesn't seem reusable between different builds.

Is there any classification mechanism that doesn't depend on all the possible inputs and build args and only the ones that could affect compiled binary compatibility (such as the target CPU architecture, or -shared)?

I am looking for the best way to preserve compiled modules dependencies and distribute/ship them in several supported flavors (basically what I described above - CPU arch, -shared and any other flag that may affect compatibility between compiled object files; the compiled artifacts that have a matching combination of all those would be picked up automatically by the build without recompilation).

The goal is to save the build fleet time that would be spent on repeatedly recompiling popular modules from sources (most importantly the stdlib, but not limited to it).