CC @golang/runtime

This does seem impossible. Have you tried running the program under the race detector?

CC @golang/runtime

This does seem impossible. Have you tried running the program under the race detector?

Added -race to the build, but unsure if there are additional steps required, the panic remained, no race reported.

Passing on gotip tyk: devel go1.23-3776465 Fri May 24 22:23:55 2024 +0000, but panic on >=1.22.0 <=1.22.3

If you have a passing and a failing state, then a binary search might reveal which CL changed things.

Does your program use plugins? I noticed the -tags=goplugin build tag, which is suggestive of that.
I would not be surprised if this was a plugin bug. Getting type identity right in the presence of plugins is tricky.

If you have a passing and a failing state, then a binary search might reveal which CL changed things.

👍 - is there perhaps a guide we could follow, or do we just list the git hashes and have at it?

Does your program use plugins? I noticed the -tags=goplugin build tag, which is suggestive of that. I would not be surprised if this was a plugin bug. Getting type identity right in the presence of plugins is tricky.

We do use plugins, however no plugins are loaded as part of the test (no .so files, no plugin.Open). Likely we can disable CGO (and disable plugins) and see if the issue persists too.

You can use git bisect https://git-scm.com/docs/git-bisect
I have to relearn it each time I use it :(
At each stage you'll have to run make.bash in the Go repository and then build/run your test with the result.

@randall77 one of our brilliant SRE guys managed to do as you suggested:

• CGO_ENABLED=0 build still causes the panic,
• the bisect seems to point to the fix to:
  • https://go-review.googlesource.com/c/go/+/567335
  • b8c76ef

In light of the traced CL, is there some work around to the behaviour, or some indication to what may be causing it? As mentioned, a direct build from source seems to be passing, so there should be some wider environment difference either at build or run time that result in the panic, meaning there should be some way to avoid it...

the bisect seems to point to the fix to: https://go-review.googlesource.com/c/go/+/567335

Excellent, thanks. That CL certainly looks related.
I will investigate later today. Maybe we can roll that CL back, although it was fixing a different bug.

I will investigate later today. Maybe we can roll that CL back, although it was fixing a different bug.

Never mind, that CL fixes the problem. So I guess we could reconsider backporting (which we chose not to do).
It is strange that your failure is only on 1.22, where #65957 was an issue since at least 1.18.

Both builds are built from the same source tree, but build from source doesn't trigger the panic.

I'm afraid I don't understand this. What other build is there? You mention goreleaser, but I don't know what that is.

is there some work around to the behaviour

Maybe. Using the type [2]string in just the right package (go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal?) might help.

It is strange that your failure is only on 1.22, where #65957 was an issue since at least 1.18.

Even stranger is that it's only a failure if built via goreleaser (release tooling). If built from source without the indirection, via Dockerfile, then the failure doesn't appear. Also doesn't appear on 1.21 with same goreleaser release pipeline.

I've asked if we could pinpoint the breaking CL as well.

I'm afraid I don't understand this. What other build is there? You mention goreleaser, but I don't know what that is.

We build two different processes as I tried to describe in the issue:

• a local dockerfile, built directly with go build, which passes the test suite even with 1.22.3
• a release CI action which uses goreleaser for building and packaging, docker installed via deb (task test:build for a local partial build, particular for amd64)

goreleaser is in essence release build tooling that provides a wrapper around go build, creating deb and rpm packages, and ultimately creating a docker image where those packages are installed ; our release build is failing those CI tests, however the very minimal dockerfile that skips all of those steps and just uses go build is passing the CI tests.

Maybe. Using the type [2]string in just the right package (go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal?) might help.

I didn't catch you there, the key is a type struct{ptr, ptr}, and it's just in an imported third party package, type safety shouldn't enable using [2]string in lieu of the code that lives there.

Hm, I'm not sure what goreleaser might do differently then. Certainly trying to match what goreleaser does in your simple docker build, or paring back what goreleaser does to match the simple docker build, might illuminate things.

One thing I would check: make sure that you're actually getting the right Go version in both cases. You can print runtime.Version() into a log somewhere, or run go version <binary> on the binary in the final docker file.

I didn't catch you there, the key is a type struct{ptr, ptr}, and it's just in an imported third party package, type safety shouldn't enable using [2]string in lieu of the code that lives there.

I mean doing, anywhere in the package, something like:

var a any = [2]string{}
var b any = [2]string{}
func init() {
    if a != b {
        panic("bad")
    }
}

This just introduces a use of [2]string, and in a context where equality must work, into the package.

This would be the breaking CL: cf68384
Title: cmd/compile/internal/gc: steps towards work-queue

One thing I would check: make sure that you're actually getting the right Go version in both cases. You can print runtime.Version() into a log somewhere, or run go version on the binary in the final docker file.

Verified with go version -m before filing the issue, output between failing and passing binary matches 1-1. ☑️

Just want to add we are seeing this as well. Same runtime panic within the opentelemetry code.

go 1.22.4 on MacOS 14.5 Sonoma with M1 Max chip.

go env output

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/kellen.miller/Library/Caches/go-build'
GOENV='/Users/kellen.miller/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/kellen.miller/go/pkg/mod'
GOOS='darwin'
GOPATH='/Users/kellen.miller/go'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.22.4/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.22.4/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.4'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/69/3t8bmcl909g3zc07kv0v1dyr0000gr/T/go-build3499618249=/tmp/go-build -gno-record-gcc-switches -fno-common'

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

This isn't actually done yet, reopening.
I'm presuming this is a dup of #65957, so it should be fixed with the next point release (1.22.5).