Change https://go.dev/cl/473115 mentions this issue: cmd/link: don't warn about llvm generated __covrec_ symbols

Where it it created for future reference: https://github.com/llvm/llvm-project/blob/3bc76339ea887e903d3e72f0e973d117d9c69c35/clang/lib/CodeGen/CoverageMappingGen.cpp#L1643

Thanks for the report.

It would be helpful if you could post the C flags that are being used for this build (e.g. what's being passed to the C compiler when the various CGO C files are compiled).

Also, I am curious as to what happens if you switch to external linking (e.g. "-ldflags=-linkmode=external"), whether that also resolve the issue.

Thank you for the quick reply!

    CGO_CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -fno-omit-frame-pointer -Wthread-safety -Wself-assign --target=x86_64-linux-gnu -fPIC -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fdebug-prefix-map=external/llvm_toolchain/=external/llvm_toolchain/ --sysroot=external/foo_sysroot_k8s -fprofile-instr-generate -fcoverage-mapping' \
    CGO_ENABLED=1 \
    CGO_LDFLAGS='-lm -no-canonical-prefixes --target=x86_64-linux-gnu -fuse-ld=lld -Lexternal/llvm_toolchain//lib -l:libc++.a -l:libc++abi.a -l:libunwind.a -rtlib=compiler-rt -lpthread -ldl -Wl,--build-id=md5 -Wl,--hash-style=gnu -Wl,-z,relro,-z,now -Wl,--unresolved-symbols=ignore-in-object-files -Lexternal/foo_sysroot_k8s/usr/lib/x86_64-linux-gnu --sysroot=external/foo_sysroot_k8s -fprofile-instr-generate' \
    GOARCH=amd64 \
    GOOS=linux \

I belive that -fprofile-instr-generate -fcoverage-mapping is the ones that creates the coverage symbols.

I'll see if I can convince bazel to run with -linkmode=external :)

It looks like passing -linkmode=external doesn't do any difference, still fails.
Would it make sense to omit the __covrec_ symbols in the elf loader as per my CL above?

@gopherbot, please remove label WaitingForInfo

Thanks. The failure with -linkmode=external is surprising -- would you mind posting the error output for that?

Yeah, that was a bit surprising to me as well. I get the same error output from loadelf so I supposed it tries to load the elf before invoking the external linker?

The rules_go GoLink action step w/o -linkmode=external:

  exec env - \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GOOS=linux \
    GOPATH='' \
    GOROOT=bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_ \
    GOROOT_FINAL=GOROOT \
    PATH=external/llvm_toolchain/bin:/bin:/usr/bin \
  bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/builder_reset/builder '-param=bazel-out/k8-fastbuild/bin/foo/bar/service/common/go_default_test_/go_default_test-0.params' -- -extld external/llvm_toolchain/bin/clang -s -w -X 'github.com/bazelbuild/rules_go/go/tools/bzltestutil.RunDir=foo/bar/service/common' '-buildid=redacted' -extldflags '-lm -no-canonical-prefixes --target=x86_64-linux-gnu -fuse-ld=lld -Lexternal/llvm_toolchain//lib -l:libc++.a -l:libc++abi.a -l:libunwind.a -rtlib=compiler-rt -lpthread -ldl -Wl,--build-id=md5 -Wl,--hash-style=gnu -Wl,-z,relro,-z,now -Wl,--unresolved-symbols=ignore-in-object-files -Lexternal/my_sysroot_k8s/usr/lib/x86_64-linux-gnu --sysroot=external/my_sysroot_k8s -fprofile-instr-generate --coverage')

And with -linkmode=external it is slightly different:

exec env - \
    CC=external/llvm_toolchain/bin/clang \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GOOS=linux \
    GOPATH='' \
    GOROOT=bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_ \
    GOROOT_FINAL=GOROOT \
    PATH=external/llvm_toolchain/bin:/bin:/usr/bin \
  bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/builder_reset/builder '-param=bazel-out/k8-fastbuild/bin/foo/bar/service/service.bin-0.params' -- -extld external/llvm_toolchain/bin/clang '-buildid=redacted' -extldflags '-lm -no-canonical-prefixes --target=x86_64-linux-gnu -fuse-ld=lld -Lexternal/llvm_toolchain//lib -l:libc++.a -l:libc++abi.a -l:libunwind.a -rtlib=compiler-rt -lpthread -ldl -Wl,--build-id=md5 -Wl,--hash-style=gnu -Wl,-z,relro,-z,now -Wl,--unresolved-symbols=ignore-in-object-files -Lexternal/my_sysroot_k8s/usr/lib/x86_64-linux-gnu --sysroot=external/my_sysroot_k8s -fprofile-instr-generate --coverage')

Hmm, based on what I am seeing I think the "ldflags=-linkmode=external" is maybe not making it through to the Go command... the reason I say this is that the "loadelf" routines in question (which are generating the errors) are only invoked during internal linking.

Here's a toy program that uses the os/user package:

package main

import "os/user"

func main() {
	if g, err := user.LookupGroup("mumble"); err != nil {
		println("that was susprising", g != nil)
	}
}

and here is a script I can use to compile it. Note the use of -fprofile-instr-generate -fcoverage-mapping etc.

#!/bin/sh
set -x
rm -f toy.exe
CC=clang-14 \
CGO_CFLAGS="-fstack-protector -fno-omit-frame-pointer -Wthread-safety -Wself-assign --target=x86_64-linux-gnu -fPIC -no-canonical-prefixes -Wno-builtin-macro-redefined -fprofile-instr-generate -fcoverage-mapping" \
  CGO_ENABLED=1 \
  CGO_LDFLAGS="-lm -no-canonical-prefixes --target=x86_64-linux-gnu -fuse-ld=lld -rtlib=compiler-rt -lpthread -ldl -Wl,--unresolved-symbols=ignore-in-object-files -fprofile-instr-generate" \
    GOARCH=amd64 \
    GOOS=linux \
    go build -o toy.exe $* toy.go

If I do a build with "-ldflags=-linkmode=external" this works fine (external linker can handle the sections generated by the coverage flags), whereas if I build with "-ldflags=-linkmode=internal" I get the same sorts of errors that you are seeing with your bazel build. Eg

loadelf: $WORK/b001/_pkg_.a(_x002.o): 53157: sym#5 (__profc_sayit): ignoring symbol in section 14 (type 0)
loadelf: $WORK/b003/_pkg_.a(_x002.o): 53161: sym#4 (__covrec_3BF569B9C342ECE1): ignoring symbol in section 4 (type 1)
loadelf: $WORK/b003/_pkg_.a(_x003.o): 53166: sym#4 (__profc__cgo_release_context): ignoring symbol in section 7 (type 0)
loadelf: $WORK/b003/_pkg_.a(_x004.o): 53174: sym#6 (__profc_fatalf): ignoring symbol in section 8 (type 0)
loadelf: $WORK/b003/_pkg_.a(_x005.o): 53196: sym#10 (__profc_x_cgo_sys_thread_create): ignoring symbol in section 19 (type 0)
...

See also related issues #58619 and #58620, similar things are happening there.

The best fix for all these problems is to make sure that we wind up using the external linker, as opposed to continuing to use the internal linker and teaching it to deal with the various quirks in host objects.

I am working on a CL 471055 that will help solve these problems using the Go command, however in your case you are building with bazel, so we either need a bazel fix or we need some other fix in the Go toolchain (possibly the cgo command).

Hmm, based on what I am seeing I think the "ldflags=-linkmode=external" is maybe not making it through to the Go command... the reason I say this is that the "loadelf" routines in question (which are generating the errors) are only invoked during internal linking.

Interesting, as a simple test I did change to -linkmode=foo and that makes it error out with the link help message, so it gets through in some way. Thanks for confirming that loadelf isn't invoked during internal linking, I'll see if I can find out why the external linkmode doesn't bite in my case.

I am working on a CL 471055 that will help solve these problems using the Go command, however in your case you are building with bazel, so we either need a bazel fix or we need some other fix in the Go toolchain (possibly the cgo command).

Yeah, as far as I've seen rules_go uses go test -cover and doesn't care about llvm coverage so I guess it could strip llvm coverage flags before compiling? It doesn't really feel like the cleanest solution though 😅

Change https://go.dev/cl/475375 mentions this issue: cmd/go,cmd/link: prefer external linking when strange cgo flags seen

Update: I have submitted CL 475375, which should take care of the problem for "go test". If you can test this CL (or latest tip of go repo) to see if it fixes the problem, that would be great.

Super! I tested with go version devel go1.21-035db07d7c Tue Mar 14 13:17:46 2023 +0000 linux/amd64 and it do indeed fix this issue, thanks!

Are there any slim chances that CL 475375 gets back ported to 1.20? As it seems to prevent others from updating to Go 1.20 too.

Thanks for verifying.

I'll request a backport-- it will need to be reviewed/approved by the release team, however, so no guarantees.

Change https://go.dev/cl/476576 mentions this issue: cmd/go,cmd/link: prefer external linking when strange cgo flags seen

Change https://go.dev/cl/476577 mentions this issue: cmd/go,cmd/link: prefer external linking when strange cgo flags seen

Closing out this issue, I don't believe there is anything remaining to do here.