cmd/link: missing CGO symbols in `.dynsym` on Go >= 1.21 on arm64 #62520

mtardy · 2023-09-08T10:01:05Z

What version of Go are you using (`go version`)?

$ go version
go version go1.21.1 linux/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/home/mahe.linux/.cache/go-build'
GOENV='/home/mahe.linux/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/mahe.linux/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/mahe.linux/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.21.1'
GCCGO='gccgo'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1359563043=/tmp/go-build -gno-record-gcc-switches'

What did you do?

So the issue is a difference of behavior from Go 1.20 to Go 1.21 on arm64 (and not on amd64).

Let's make a minimal reproducing setup:

Put that in a file, for example main.go and go build main.

//go:build linux
// +build linux

package main

/*
int symbol_test(void)
{
        return 0;
}
*/
import "C"

func main() {
        C.symbol_test()
}

Use a binary to read the symbols, for example readelf -s main | grep symbol_test, you can use less as well to see:
- With Go 1.20.8 (and before), there is a symbol_test in .dynsym and symbol_test in .symtab.
- With Go 1.21.0 and 1.21.1, there is only a symbol_test in .symtab, it was removed from .dynsym.

So I cloned Go and did a git bissect to see which commit was the culprit and found that it was 1f29f39 -> https://go-review.googlesource.com/c/go/+/414654. Unfortunately, I lack context and understanding of the Go codebase to understand why this changed the behavior like that on arm64 specifically and if this is unintended.

For context, I was using uprobes on a Go binary, and for a reason that is not entirely clear for now, the lib I was using to load the probe is only reading .dynsym.

What did you expect to see?

I expected this behavior not to change, and especially not to change only on arm64 while amd64 is still behaving the same as in the past.

What did you see instead?

I saw that the symbol was missing in .dynsym using Go >= 1.21.

The text was updated successfully, but these errors were encountered:

seankhliao · 2023-09-08T13:01:56Z

cc @ianlancetaylor

cherrymui · 2023-09-08T15:28:26Z

Thanks for the report. Previously we dynamically export ~all symbols, which is too many and unnecessary for many of them. With that CL, we only export Go symbols that are dynamically exported to C. I think we probably should also export C symbols (that are not marked as static or hidden visibility) per C semantics. @ianlancetaylor what do you think?

cherrymui · 2023-09-08T15:30:23Z

on arm64 (and not on amd64)

This is interesting. I don't think that behavior depends on the architecture. Maybe this could be due to different C toolchain versions? What C compiler and linker do you use for both architectures?

mtardy · 2023-09-08T16:34:44Z

on arm64 (and not on amd64)

This is interesting. I don't think that behavior depends on the architecture. Maybe this could be due to different C toolchain versions? What C compiler and linker do you use for both architectures?

Yes that may be the reason. Initially, I uncovered that new thing because in the PR to update to Go 1.21 for the project I'm working on cilium/tetragon#1344, we had only the Go test failing on arm64. See this run for example https://github.com/cilium/tetragon/actions/runs/6119884977/job/16610682208?pr=1344. The reason why it failed is because we load a uprobe at some point and a symbol that we previously used was missing only on the arm64 runners binaries. But those are GitHub-hosted runners for x86_64 and BuildJet self-hosted runners for arm64, so they might have different toolchains installed indeed (and I can't tell you exactly what's running on those without running workflows for that). I could however reproduce easily the issue to isolate the commit on my arm64 machine.

ianlancetaylor · 2023-09-08T20:22:43Z

The C linker doesn't export all symbols to the dynamic symbol table by default. I think that people who rely on this should be passing -extldflags=-Wl,--export-all-symbols to go build. It should also work to add this to a #cgo LDFLAGS line in a cgo comment. Instead of --export-all-symbols you could of course use --export-dynamic-symbol for particular symbols.

In general CL 414654 made the Go linker behave more like a C linker with regard to when dynamic symbols are exported. I think that is a good thing and I don't yet see a reason to change that. I agree that it is a change in behavior.

mtardy · 2023-09-11T09:50:33Z

So I just understood something new, my issue originally came from the fact that when compiling and running with go test (without the -c flag), copying the binary and readelf(ing) will end up with a binary missing the .symtab (thus being the only one containing the symbol in Go 1.21, which means the symbol cannot be found), only .dynsym is there. When I do go test -c, the .symtab is there, and the test passes because I can find the symbol (in .symtab).

Basically, when I do go test, I end up with a stripped binary, and with go test -c the binary is not stripped. I thought those were equivalent and go test was just a shortcut to compiling and running.

I'm not very familiar with how behaves go test but why do I have this difference?

cherrymui · 2023-09-11T16:21:03Z

go test without -c generates a one-off binary that is supposed to immediately run and get deleted, so it by default suppresses symbol table, debug info, etc. to make build faster. It is not a binary that you want to copy or hold for it. If you need a binary to investigate, use go test -c, which generates a binary with symbol table and debug info.

mtardy · 2023-09-11T16:23:24Z

go test without -c generates a one-off binary that is supposed to immediately run and get deleted, so it by default suppresses symbol table, debug info, etc. to make build faster. It is not a binary that you want to copy or hold for it. If you need a binary to investigate, use go test -c, which generates a binary with symbol table and debug info.

ah yeah ok I see. I end up with a different behavior since I'm using the binary itself to hook a uprobe and then it can now fail since the symbol is now absent from .dynsym and the whole .symtab can be removed as well. But I should probably not do that and rely on a separate program, thanks for all your answers :)

cherrymui · 2023-09-11T16:31:49Z

Thanks. Is there anything still needed for this issue? Or we can close it? Thanks.

mtardy · 2023-09-11T18:16:38Z

Thanks. Is there anything still needed for this issue? Or we can close it? Thanks.

Cool! I guess no regarding this response:

In general CL 414654 made the Go linker behave more like a C linker with regard to when dynamic symbols are exported. I think that is a good thing and I don't yet see a reason to change that. I agree that it is a change in behavior.

If the change of behavior is okay/better on your side, I just need to adapt and it won't be hard. I was just thinking that it might have been unexpected for the Go project and was interested in it.

I was going to close it but I'll let you close it as not planned or completed, whatever you prefer.

erifan · 2023-09-12T01:27:58Z

on arm64 (and not on amd64).

This seems to be true.
On my Linux/arm64 machine:

$ go build -o main2 main2.go
$ readelf -s main2 | grep symbol_test
  1780: 0000000000459e90     8 FUNC    GLOBAL DEFAULT   14 symbol_test

$ go env

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/home/erifan01/.cache/go-build'
GOENV='/home/erifan01/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/erifan01/gopath/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/erifan01/gopath'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/erifan01/go-master'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/erifan01/go-master/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='devel go1.22-5eb382fc08 Sun Sep 10 23:21:34 2023 +0000'
GCCGO='/usr/bin/gccgo'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3909050461=/tmp/go-build -gno-record-gcc-switches'

gcc version 13.1.0 (Ubuntu 13.1.0-8ubuntu1~22.04)

On my Linux/amd64 machine:

$ go build -o main2 main2.go
$ readelf -s main2 | grep "symbol_test"
    61: 000000000045ab30     7 FUNC    GLOBAL DEFAULT   14 symbol_test
  1174: 000000000045aaa0    84 FUNC    LOCAL  DEFAULT   14 main._Cfunc_symbol_test.a
  1723: 000000000045ab30     7 FUNC    GLOBAL DEFAULT   14 symbol_test

$go env

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/erifan02/.cache/go-build'
GOENV='/home/erifan02/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/erifan02/gopath/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/erifan02/gopath'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/erifan02/go-master'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/erifan02/go-master/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.22-5eb382fc08 Sun Sep 10 23:21:34 2023 +0000'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3754892114=/tmp/go-build -gno-record-gcc-switches'

gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)

I don't know if this is caused by the difference in GCC version, but this difference does exist.

mtardy · 2023-09-14T14:37:36Z

I guess this could use a little bit more effort to make sure behavior is the same with different architecture (and same everything, Go, gcc) and thus might be reopened if this is a real thing (on my side I still experience the architecture difference).

cherrymui · 2023-09-14T15:23:35Z

I think this is conditioned on whether the C linker supports the --export-dynamic-symbol flag. If it doesn't support this flag we'll (have to) export all symbols. Can you try to build a trivial C program with gcc -Wl,--export-dynamic-symbol=main to see if the C toolchain on both machines support the flag?

erifan · 2023-09-15T01:44:05Z

On my Linux/amd64 machine, the C linker does not support the --export-dynamic-symbol option.

$ gcc -Wl,--export-dynamic-symbol=main main.c
/usr/bin/ld: unrecognized option '--export-dynamic-symbol=main'
/usr/bin/ld: use the --help option for usage information
collect2: error: ld returned 1 exit status

$ ld -v
GNU ld (GNU Binutils for Ubuntu) 2.34

On my Linux/arm64 machine, the C linker supports this option.

cherrymui · 2023-09-15T15:42:27Z

Thanks. So this explains the difference.

ianlancetaylor · 2023-09-15T19:07:29Z

The GNU linker picked up the --export-dynamic-symbol option in 2.35, released in 2020.

Go 1.21 broke dynamic symbol linking export, see more - golang/go#62520 - golang/go#62520 (comment) Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com>

bcmills added the compiler/runtime Issues related to the Go compiler and/or runtime. label Sep 8, 2023

seankhliao changed the title ~~linker: missing CGO symbols in .dynsym on Go >= 1.21 on arm64~~ cmd/link: missing CGO symbols in .dynsym on Go >= 1.21 on arm64 Sep 8, 2023

cherrymui added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Sep 8, 2023

cherrymui added this to the Go1.22 milestone Sep 8, 2023

cherrymui added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 11, 2023

ianlancetaylor closed this as not planned Won't fix, can't repro, duplicate, stale Sep 12, 2023

mtardy mentioned this issue Sep 14, 2023

chore(deps): update go to v1.21.1 (minor) cilium/tetragon#1344

Merged

1 task

mtardy added a commit to cilium/tetragon that referenced this issue Sep 28, 2023

pkg/sensors: uprobe test fix using dyn export

76f86c9

Go 1.21 broke dynamic symbol linking export, see more - golang/go#62520 - golang/go#62520 (comment) Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com>

mtardy added a commit to cilium/tetragon that referenced this issue Sep 28, 2023

pkg/sensors: uprobe test fix using dyn export

b387c46

Go 1.21 broke dynamic symbol linking export, see more - golang/go#62520 - golang/go#62520 (comment) Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/link: missing CGO symbols in `.dynsym` on Go >= 1.21 on arm64 #62520

cmd/link: missing CGO symbols in `.dynsym` on Go >= 1.21 on arm64 #62520

mtardy commented Sep 8, 2023

seankhliao commented Sep 8, 2023

cherrymui commented Sep 8, 2023

cherrymui commented Sep 8, 2023

mtardy commented Sep 8, 2023

ianlancetaylor commented Sep 8, 2023

mtardy commented Sep 11, 2023 •

edited

cherrymui commented Sep 11, 2023

mtardy commented Sep 11, 2023

cherrymui commented Sep 11, 2023

mtardy commented Sep 11, 2023

erifan commented Sep 12, 2023

mtardy commented Sep 14, 2023

cherrymui commented Sep 14, 2023

erifan commented Sep 15, 2023

cherrymui commented Sep 15, 2023

ianlancetaylor commented Sep 15, 2023

cmd/link: missing CGO symbols in .dynsym on Go >= 1.21 on arm64 #62520

cmd/link: missing CGO symbols in .dynsym on Go >= 1.21 on arm64 #62520

Comments

mtardy commented Sep 8, 2023

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

seankhliao commented Sep 8, 2023

cherrymui commented Sep 8, 2023

cherrymui commented Sep 8, 2023

mtardy commented Sep 8, 2023

ianlancetaylor commented Sep 8, 2023

mtardy commented Sep 11, 2023 • edited

cherrymui commented Sep 11, 2023

mtardy commented Sep 11, 2023

cherrymui commented Sep 11, 2023

mtardy commented Sep 11, 2023

erifan commented Sep 12, 2023

mtardy commented Sep 14, 2023

cherrymui commented Sep 14, 2023

erifan commented Sep 15, 2023

cherrymui commented Sep 15, 2023

ianlancetaylor commented Sep 15, 2023

cmd/link: missing CGO symbols in `.dynsym` on Go >= 1.21 on arm64 #62520

cmd/link: missing CGO symbols in `.dynsym` on Go >= 1.21 on arm64 #62520

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

mtardy commented Sep 11, 2023 •

edited