Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/link: regression since 1.22 arm7 binary fails to start on 4.19.163 kernel with could not verify signature of ./main #72866

Open
Jorropo opened this issue Mar 14, 2025 · 12 comments
Open

cmd/link: regression since 1.22 arm7 binary fails to start on 4.19.163 kernel with could not verify signature of ./main #72866

Jorropo opened this issue Mar 14, 2025 · 12 comments
Labels
BugReport Issues describing a possible bug in the Go implementation. compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@Jorropo
Copy link
Member

Jorropo commented Mar 14, 2025 •
edited
Loading

Go version

go version go1.24.0 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='0'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm'
GOARM='7'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -marm -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build161984771=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.0'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Source:

package main

import "fmt"

func main() {
    fmt.Println("Hallo")
}

Build command:

GOARCH=arm GOARM=7 GOOS=linux go build main.go

Built in Docker official image golang1.24:

Linux linux 6.8.0-55-generic #57-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 12 23:42:21 UTC 2025 x86_64 GNU/Linux

What did you see happen?

Running in NetModule linux image:

Linux 0000-002 4.19.163 #1 SMP Fri Nov 10 10:54:34 UTC 2023 armv7l GNU/Linux

Running on Hardware:

NB2810 - https://www.netmodule.com/de/produkte/router/nb2810-2lwac-g

Exec

root@0000-002:~/gotest# ./main
Killed

dmesg

[    0.000000] could not verify signature of ./main

What did you expect to see?

root@0000-002:~/gotest# ./main
Hallo
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Mar 14, 2025
@Jorropo
Copy link
Member Author

Jorropo commented Mar 14, 2025

Reported by icsharp. in discord gophers: https://discord.com/channels/118456055842734083/1344643961246978118/1344643961246978118

@Jorropo
Copy link
Member Author

Jorropo commented Mar 14, 2025

Works if the target hardware is running kernel 5.15.147 #1 SMP Fri Jul 19 14:30:48 UTC 2024 armv7l armv7l armv7l GNU/Linux

@Jorropo
Copy link
Member Author

Jorropo commented Mar 14, 2025

Bisected to 3251006 cc @thanm

@Jorropo Jorropo added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 14, 2025
@Jorropo
Copy link
Member Author

Jorropo commented Mar 14, 2025 •
edited
Loading

Surprisingly this works if built with CGO:

GOOS=linux GOARCH=arm CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc go build  -ldflags "-linkmode=external -extldflags=-static" main.go

@Jorropo Jorropo changed the title cmd/link: regression since 1.22 arm7 binary fails to start on 4.19.163 kernel with could not verify signature of ./main` cmd/link: regression since 1.22 arm7 binary fails to start on 4.19.163 kernel with could not verify signature of ./main Mar 14, 2025
@gabyhelp
Copy link

Related Issues

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@gabyhelp gabyhelp added the BugReport Issues describing a possible bug in the Go implementation. label Mar 14, 2025
@thanm
Copy link
Contributor

thanm commented Mar 14, 2025

4.19.163 is an ancient kernel, I am not sure it really makes sense to try to pursue a fix for something that old.

Also, the change in question is fixing an important bug. I am not sure why this particular elderly kernel would object to the new way of doing things, but the old way was definitely broken -- putting the ".shstrtab" section in a loadable segment (as we were doing before the CL) is non-standard and is not the way C compilers generate ELF files.

@Jorropo
Copy link
Member Author

Jorropo commented Mar 14, 2025 •
edited
Loading

@thanm #67001 recently moved the go1.24 baseline to 3.2 (edit 3.17), what does it mean to announce a minimum linux release if we are then gonna reconsider any bug impacting old but above the minimum kernels through an is it really worth our time to fix this lense ?

Note: I'm being devil's advocate here.

@prattmic
Copy link
Member

prattmic commented Mar 14, 2025 •
edited
Loading

Do we know if this bug is general to any 4.19 kernel, or some custom change this vendor made to their kernels?

I checked out a copy of the 4.19 kernel source and can't find the string "could not verify signature" at all. (Though admittedly I didn't look very hard).

I'm at a bit of a loss what signature it would be verifying in the first place? Linux binaries aren't signed.

@HESS-BEA
Copy link

Hi, im icsharp.

First of all, i can't not say im sure they did no changes to the kernel...

Is there any way i could make more tests or something like that?
Im indirect in contact with the PO of them, i will ask more questions on monday, maybe he can tell me more.

Something specific i could ask?

@prattmic
Copy link
Member

I'd ask them what the verification is, and whether they can test Go binaries themselves.

I tested a Debian Buster image from https://cdimage.debian.org/images/cloud/, which also uses a 4.19 kernel, and Go 1.24 binaries ran just fine. Though this was on amd64, not arm.

@ianlancetaylor
Copy link
Member

@HESS-BEA To the best of my knowledge that error message must be coming from a non-standard authentication module of some sort, probably via Linux-PAM. Look at /etc/pam.conf and /etc/pam.d. It's very unlikely that this is a part of the Linux kernel proper. I would guess that this is some sort of authentication installed by whoever built the router software.

If you can find the PAM module that is requiring the signature, then perhaps you can find out what will satisfy it.

@Jorropo Are you able to recreate the problem, or are you passing on data from Discord? Either way, if it works with -linkmode=external, that means that the C linker is either producing the required signatures or is generating an executable that somehow does not require a signature.

@Jorropo
Copy link
Member Author

Jorropo commented Mar 14, 2025 •
edited
Loading

Are you able to recreate the problem, or are you passing on data from Discord?

Passing data from discord.

that means that the C linker is either producing the required signatures or is generating an executable that somehow does not require a signature

This is cross compiled in a default go docker so producing any signature seems unlikely.
Because it works using CGO I thought it might be a missconfigured linker option someone experienced there would find quickly altho I did not considered something like:

I checked out a copy of the 4.19 kernel source and can't find the string "could not verify signature" at all. (Though admittedly I didn't look very hard).

@mknyszek mknyszek added this to the Backlog milestone Mar 19, 2025
@mknyszek mknyszek added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
BugReport Issues describing a possible bug in the Go implementation. compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Go Compiler / Runtime
Status: Todo
Milestone
Backlog
Development

No branches or pull requests
8 participants
@mknyszek @prattmic @ianlancetaylor @gopherbot @thanm @Jorropo @HESS-BEA @gabyhelp