Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: 1.23 hangs when running under qemu-user [bisected] #70329

Open
cheese opened this issue Nov 14, 2024 · 4 comments
Open

runtime: 1.23 hangs when running under qemu-user [bisected] #70329

cheese opened this issue Nov 14, 2024 · 4 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unplanned

Comments

@cheese
Copy link

cheese commented Nov 14, 2024 •
edited
Loading

Go version

go version go1.23.3 linux/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.23.3'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2732334604=/tmp/go-build -gno-record-gcc-switches'

What did you do?

When I build Go 1.23 with the same version of Go under qemu-user, the build hangs all the way. The issue can be reproduced with a Dockerfile:

from quay.io/fedora/fedora:41

RUN dnf install -y @development-tools
RUN curl -O https://go.dev/dl/go1.23.3.src.tar.gz && curl -O https://go.dev/dl/go1.23.3.linux-arm64.tar.gz
RUN tar fx go1.23.3.linux-arm64.tar.gz && mkdir src && tar fx go1.23.3.src.tar.gz  -C src
RUN cd src/go/src/; env PATH=$PATH:/go/bin bash make.bash

Steps:

  1. Setup qemu-user binfmt for a foreign ISA, for example, installs qemu-user-static-aarch64 on Fedora.
  2. Build the Dockerfile for specified arch: podman build --arch aarch64 .

Tested on Fedora 40 with 6.9.8-200.fc40.x86_64 and Fedora 41 with 6.11.6-300.fc41.x86_64.

Tested with qemu-user 9.1.1 of riscv64/loongarch64/aarch64.

What did you see happen?

The go processes of the build hangs forever:

                 │ ├─crun-buildah-buildah663882417.scope
                 │ │ └─container
                 │ │   ├─912459 /usr/bin/qemu-aarch64-static /usr/bin/bash make.bash
                 │ │   ├─914348 /usr/bin/qemu-aarch64-static ./cmd/dist/dist bootstrap -a
                 │ │   ├─914395 /usr/bin/qemu-aarch64-static /go/bin/go install "-tags=math_big_pure_go compiler_bootstrap purego" bootstrap/cmd/...
                 │ │   └─914743 /usr/bin/qemu-aarch64-static /go/bin/go install "-tags=math_big_pure_go compiler_bootstrap purego" bootstrap/cmd/...

The issue does not exist with Go 1.22. I did a bisect and found the offending commit d068c2c .

What did you expect to see?

Build of Go 1.23 runs through under qemu-user.
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Nov 14, 2024
@gabyhelp
Copy link

Related Issues

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@cherrymui
Copy link
Member

Please note that qemu-user is not really a supported problem. As it is caused by CL 560615, it is probably due to a qemu bug or inconsistency in how it handles kernel APIs.

cc @panjf2000 @ianlancetaylor for CL 560615

@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 15, 2024
@cherrymui cherrymui added this to the Unplanned milestone Nov 15, 2024
@cheese
Copy link
Author

cheese commented Dec 21, 2024

Cross-posted to qemu: https://gitlab.com/qemu-project/qemu/-/issues/2738

@cheese
Copy link
Author

cheese commented Dec 23, 2024 •
edited
Loading

Limiting GOMAXPROCS to a small number can relieve this issue.

SalDaniele added a commit to SalDaniele/cluster-deployment-automation that referenced this issue Feb 18, 2025
@SalDaniele
Add workaround to prevent go build from hanging
Loading
Loading status checks…
53da7bb 
There is an issue that causes go multiarch to hang indefinitely sometimes: golang/go#70329

Per the comments on the issue, it seems setting GOMACPROCS to a small number can relieve this issue as a temporary WA

Signed-off-by: Salvatore Daniele <sdaniele@redhat.com>
@SalDaniele SalDaniele mentioned this issue Feb 18, 2025
Merged
SalDaniele added a commit to SalDaniele/cluster-deployment-automation that referenced this issue Feb 18, 2025
@SalDaniele
Add workaround to prevent go build from hanging
Loading
Loading status checks…
432c065 
There is an issue that causes go multiarch to hang indefinitely sometimes: golang/go#70329

Per the comments on the issue, it seems setting GOMACPROCS to a small number can relieve this issue as a temporary WA

Signed-off-by: Salvatore Daniele <sdaniele@redhat.com>
SalDaniele added a commit to SalDaniele/dpu-operator that referenced this issue Feb 19, 2025
@SalDaniele
Add gomaxproc when building
dd402b9 
There is an issue that causes go multiarch to hang indefinitely
sometimes [1].

Per the comments on the issue, it seems setting GOMAXPROCS to a small
number can relieve this issue as a temporary workaround

[1] golang/go#70329

Signed-off-by: Salvatore Daniele <sdaniele@redhat.com>
@SalDaniele SalDaniele mentioned this issue Feb 19, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
4 participants
@cheese @gopherbot @cherrymui @gabyhelp