cmd/go: cgo-enabled binary built with Ubuntu 22.04 is incompatible with Ubuntu 20.04 #57328

zeisss · 2022-12-15T15:16:47Z

What version of Go are you using (`go version`)?

$ ~/go/bin/go version
go version go1.19.4 linux/amd64

Does this issue reproduce with the latest release?

This is the latest stable release from the website.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ ~/go/bin/go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOENV="/home/ubuntu/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ubuntu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ubuntu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ubuntu/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/ubuntu/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build646395045=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Building a minimal binary (using net/http code) with -trimpath on Ubuntu 22.04 makes it incompatible with Ubuntu 20.04:

ubuntu@jammy:~$ cat main.go
package main

import "fmt"
import "net/http"

func main() {
	fmt.Println("Hello World")

	http.ListenAndServe(":8080", nil)
}
ubuntu@jammy:~$ ~/go/bin/go build -o main-jammy -trimpath ./main.go

Copying the resulting binary to an Ubuntu 20.04 machine yields the following error:

$ ./main-jammy
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./main-jammy)
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./main-jammy)

Dropping -trimpath from the go build call and the binary works.

What did you expect to see?

Invoking main-jammy on an Ubuntu 20.04 machine should yield Hello World and start a webserver.

What did you see instead?

$ ./main-jammy
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./main-jammy)
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./main-jammy)

Context

Github Actions is currently rolling out Ubuntu 22.04 for its ubuntu-latest tag and as a result some of our releases are becoming invalid with Ubuntu 20.0.

The text was updated successfully, but these errors were encountered:

bcmills · 2022-12-15T15:41:20Z

go version go1.19.4 linux/amd64

Is this a regression from a previous release? (Did it ever work?)

zeisss · 2022-12-15T15:59:36Z

Previously releases have been build on Ubuntu 20.04 on Github Actions (we are using the ubuntu-latest tag), so I cannot answer that. If I find time tomorrow, I might be able to test this with older releases.

thanm · 2022-12-15T16:05:36Z

@golang/compiler

bcmills · 2022-12-15T16:08:21Z

I think -trimpath is something of a red herring. Here's my hypothesis:

The problem is that your installed glibc has a symbol definition that is newer than the glibc of the machine on which you are deploying the binary.
When you build without -trimpath, the problem is masked because you link against the precompiled .a files that shipped with go1.19.4, and link against the glibc symbols that happened to be used to build that distribution's cgo dependencies.
However, when you build with -trimpath, the cache key for the precompiled files no longer matches, and you get a rebuild with your installed glibc.

We can bust the cache in other ways, too (such as by setting CGO_CFLAGS), reproducing the same symptom:

$ go1.19.4 build -o main main.go

$ nm -D main | grep pthread_attr_getstacksize
                 U pthread_attr_getstacksize@GLIBC_2.2.5

$ go1.19.4 build -trimpath -o main main.go

$ nm -D main | grep pthread_attr_getstacksize
                 U pthread_attr_getstacksize@GLIBC_2.34

$ CGO_CFLAGS='-O2 -g -D__go_unused_variable=1' go1.19.4 build -o main main.go

$ nm -D main | grep pthread_attr_getstacksize
                 U pthread_attr_getstacksize@GLIBC_2.34

bcmills · 2022-12-15T16:09:01Z

And in Go 1.20, you will see the newer symbols on all builds because we're not shipping precompiled archives at all (#47257). 😅

bcmills · 2022-12-15T17:06:38Z

The workarounds I can see for your use-case are:

Build the program on the system with the older glibc (so that it uses the older symbols).
Set CGO_ENABLED=0 to build a binary that doesn't depend on glibc at all.
Set various flags (see cmd/go: build: add -static flag #26492) to build a fully-static cgo-enabled binary.

bcmills · 2022-12-15T17:16:03Z

But, I am left a little bit puzzled about one thing: I don't expect -trimpath to actually affect the cache keys for the standard library: as of CL 380915 we should be building standard-library packages as if -trimpath were always set. 🤔

I'll file a separate issue to look into that.

bcmills · 2022-12-15T22:31:02Z

I've filed #57346 for the cache-invalidation behavior. Otherwise, I don't think there is anything we can feasibly do for this on the cmd/go side: cgo-enabled Go binaries inherently have many of the same compatibility issues as ordinary C binaries.

zeisss · 2022-12-16T10:13:40Z

Build the program on the system with the older glibc (so that it uses the older symbols).

Yeah, we will go with that. Since we control most execution environments, this is doable and easiest. I mainly reported this since this was surprising to me and non-obvious from the -trimpath description.

Thanks for looking into this so quickly!

Due to recent changes, the plugin image build has failed. The following changes has been added: 1. Fixing the command name in the DaemonSet spec 2. Pin golang to 1.20 3. Running the go build command with the right arguments 4. changing CGO_ENABLED=0 - The binary failed to execute and complains about GLIBC_2.32 not found. Running with CGO_ENABLED=0 build a binary that doesn't depend on glibc: golang/go#57328 (comment) Signed-off-by: Talor Itzhak <titzhak@redhat.com>

1.19.10 includes a change on linked libraries that's messing up our manager binary see golang/go#47257 and golang/go#57328 (issues are closed, but the error is the same)

Due to golang/go#57328, binaries built on Github's "ubuntu-latest" don't run on Ubuntu 20.04. Setting CGO_ENABLED=0 at build fixes this. Since we don't rely on CGO in this repo, this seems like the easiest fix.

By default we are relying on cgo for DNS resolution, which causes binaries built on ubuntu 22.04 to break on ubuntu 20.04 due to some changes in glibc. See golang/go#57328 This changes us to not rely on that, which causes builds to succeed.

The CircleCI go build images' toolchain depends on a version of GLIBC that is higher than the version that ships on Ubuntu 20.04. This means users trying to install the released binaries on Ubuntu 20.04 will see: ``` $ gptcli gptcli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by gptcli) gptcli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by gptcli) ``` This commit attempts to mitigate the problem by building w/ CGO_ENABLED=0 as suggested by: golang/go#57328

The CircleCI go build images' toolchain depends on a version of GLIBC that is higher than the version that ships on Ubuntu 20.04. This means users trying to install the released binaries on Ubuntu 20.04 will see: ``` $ bashgpt bashgpt: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by bashgpt) bashgpt: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by bashgpt) ``` This commit attempts to mitigate the problem by building w/ CGO_ENABLED=0 as suggested by: golang/go#57328

The CircleCI go build images' toolchain depends on a version of GLIBC that is higher than the version that ships on Ubuntu 20.04. This means users trying to install the released binaries on Ubuntu 20.04 will see: ``` $ spotsh spotsh: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by spotsh) spotsh: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by spotsh) ``` This commit attempts to mitigate the problem by building w/ CGO_ENABLED=0 as suggested by: golang/go#57328

ref golang/go#57328

zeisss changed the title ~~affected/package:~~ cmd/go: -trimpath makes binary incompatible with Ubuntu 20.04 when build on Ubuntu 22.04 Dec 15, 2022

thanm added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 15, 2022

bcmills added the Unfortunate label Dec 15, 2022

bcmills changed the title ~~cmd/go: -trimpath makes binary incompatible with Ubuntu 20.04 when build on Ubuntu 22.04~~ cmd/go: cgo-enabled binary built with Ubuntu 22.04 is incompatible with Ubuntu 20.04 Dec 15, 2022

This was referenced Dec 15, 2022

cmd/dist: build cmd with CGO_ENABLED=0, so cmd/go uses pure Go net resolver #57007

Closed

cmd/go: make cache keys for libraries in GOROOT insensitive to -trimpath #57346

Open

bcmills added the GoCommand cmd/go label Dec 15, 2022

bcmills added this to the Unplanned milestone Dec 15, 2022

bcmills closed this as not planned Won't fix, can't repro, duplicate, stale Dec 15, 2022

NicholasBlaskey mentioned this issue Feb 15, 2023

chore: upgrade Go from 1.18 -> 1.20 determined-ai/determined#5981

Merged

4 tasks

seankhliao mentioned this issue Feb 15, 2023

/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found #58550

Closed

DavidZidar mentioned this issue Mar 17, 2023

Can't run the v3.27.2 release on Debian because of glibc issue StackExchange/dnscontrol#2134

Closed

This was referenced Mar 22, 2023

automated devent deploy + skip verify on build ava-labs/hypersdk#94

Merged

Build with 20.04 ava-labs/hypersdk#113

Merged

g0rbe added a commit to elmasy-com/columbus-server that referenced this issue Mar 23, 2023

create static binary because of golang/go#57328

50b1b17

g0rbe added a commit to elmasy-com/columbus-scanner that referenced this issue Mar 31, 2023

fix for golang/go#57328

ebd45b0

ahobsonsayers mentioned this issue Apr 13, 2023

Cannot start on Ubuntu 20.04 FrenchGithubUser/Hatt#11

Open

FZambia mentioned this issue Apr 20, 2023

[bug] GLIBC not found with version 4.1.3 on ubuntu 20.04 focal centrifugal/centrifugo#644

Closed

mme mentioned this issue Apr 27, 2023

`GLIBC_2.32' not found in Gogle Colab mme/langforge#2

Closed

hollesse mentioned this issue Jun 13, 2023

Doesn't run on Debian 10 remotemobprogramming/mob#393

Closed

cloverstd mentioned this issue Jul 5, 2023

Fix incompatible in low version glibc HuolalaTech/page-spy-web#36

Merged

cvila84 mentioned this issue Jul 24, 2023

Helm Spray 4.0.12: error with /lib64/libc.so.6: version `GLIBC_2.34' not found ThalesGroup/helm-spray#83

Open

This was referenced Jul 25, 2023

PWX-30765: Updating golang, aws and gcloud sdk to fix vulnerabilities. libopenstorage/stork#1458

Merged

Cherry-pick PWX-30765: Updating golang, aws and gcloud sdk to fix vulnerabilities libopenstorage/stork#1465

Merged

roger2hk mentioned this issue Aug 2, 2023

CTFE GLIBC Error google/certificate-transparency-go#1118

Closed

ykharko mentioned this issue Aug 8, 2023

GLIBC not found nsmithuk/local-kms#58

Open

sigilioso mentioned this issue Aug 8, 2023

fix: use go 1.19 to asure compatibilty with older OS newrelic/nri-oracledb#135

Merged

heckelmann mentioned this issue Aug 30, 2023

[BUG] FlagD Binary not working on Ubuntu 20.X or Debian bullsyeye open-feature/flagd#878

Closed

candlerb mentioned this issue Sep 1, 2023

0.5.0 binary depends on recent GLIBC QubitProducts/exporter_exporter#94

Open

wkalt mentioned this issue Sep 27, 2023

Build CLI with -tags 'netgo' foxglove/foxglove-cli#128

Merged

antoninbas mentioned this issue Nov 17, 2023

antrea/antrea-ubi:v1.14.0 image is broken antrea-io/antrea#5722

Closed

tremblaysimon mentioned this issue Nov 27, 2023

Version 1.0.24 64-bit can't be installed on Ubuntu 20.04 pivotal-cf/spring-cloud-services-cli-plugin#142

Closed

macno added a commit to theoapp/theo-agent that referenced this issue Dec 3, 2023

CI: add job to test on 20.04 binary built on 22.04

2e328f4

ref golang/go#57328

macno added a commit to theoapp/theo-agent that referenced this issue Dec 3, 2023

CI: add job to test on 20.04 binary built on 22.04

e9999e2

ref golang/go#57328

macno added a commit to theoapp/theo-agent that referenced this issue Dec 3, 2023

CI: add job to test on 20.04 binary built on 22.04

99b6515

ref golang/go#57328

macno added a commit to theoapp/theo-agent that referenced this issue Dec 3, 2023

CI: add job to test on 20.04 binary built on 22.04

b76e7a1

ref golang/go#57328

golang locked and limited conversation to collaborators Dec 16, 2023

gopherbot added the FrozenDueToAge label Dec 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: cgo-enabled binary built with Ubuntu 22.04 is incompatible with Ubuntu 20.04 #57328

cmd/go: cgo-enabled binary built with Ubuntu 22.04 is incompatible with Ubuntu 20.04 #57328

zeisss commented Dec 15, 2022 •

edited

bcmills commented Dec 15, 2022

zeisss commented Dec 15, 2022

thanm commented Dec 15, 2022

bcmills commented Dec 15, 2022

bcmills commented Dec 15, 2022

bcmills commented Dec 15, 2022 •

edited

bcmills commented Dec 15, 2022

bcmills commented Dec 15, 2022

zeisss commented Dec 16, 2022

cmd/go: cgo-enabled binary built with Ubuntu 22.04 is incompatible with Ubuntu 20.04 #57328

cmd/go: cgo-enabled binary built with Ubuntu 22.04 is incompatible with Ubuntu 20.04 #57328

Comments

zeisss commented Dec 15, 2022 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

Context

bcmills commented Dec 15, 2022

zeisss commented Dec 15, 2022

thanm commented Dec 15, 2022

bcmills commented Dec 15, 2022

bcmills commented Dec 15, 2022

bcmills commented Dec 15, 2022 • edited

bcmills commented Dec 15, 2022

bcmills commented Dec 15, 2022

zeisss commented Dec 16, 2022

zeisss commented Dec 15, 2022 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

bcmills commented Dec 15, 2022 •

edited