CC @mknyszek, @golang/compiler.

This doesn't make any sense to me... That CL should be totally unrelated to this test. I'll investigate.

I can't reproduce by running the tests cmd/cgo/internal/test tests directly via go tool dist test on a gomote. I tried all variants, including the ones that fall under the ##### Testing cgo heading. Trying run.bash.

Yep, I can't reproduce at all on a linux-arm-aws gomote. I do not know how to proceed here. :(

This reminds me #61514, which I also don't understand and it magically stopped...

Yeah, that seems like exactly the same issue to me.

Possibly something relating to code or data alignment changing in the compiled binary?
(I'm thinking of something along the general lines of #60825.)

It is possible that this triggers some bug that is very sensitive to how the binary is laid out. For #61514, I couldn't reproduce on gomote or anything other than pre-/post-submit builds, and the test doesn't print anything besides segmentation fault, which makes debugging very hard, though.

@mknyszek, looking back at successful runs (https://build.golang.org/log/4cad61d2d6f9b3b16aa53609780318e36b984a8c) to work around #51050, I see some environment variables that might be relevant:
"GOARM=6" "GOHOSTARCH=arm" "CGO_CFLAGS=-march=armv6" "CGO_LDFLAGS=-march=armv6" (set from here).

Maybe the difficultly reproducing the problem with go tool dist test comes down to an environment setting?

Explicitly setting the environment variables didn't help. IIRC, the buildlet does set them correctly for the gomote anyway.

Here was what I ran:

gomote run -e GOARM=6 -e GOHOSTARCH=arm -e CGO_CFLAGS=-march=armv6 -e CGO_LDFLAGS=-march=armv6 go/bin/go tool dist test cmd/cgo/internal/test:static

I also tried the other cmd/cgo/internal/test test variants and run.bash. I then made sure make.bash was run with these flags set and then tried the tests again. No dice. :(

Aaaaaand it magically stopped happening again...

Ah! The failing invocation was the third invocation of cmd/cgo/internal/test, which I think makes it this one (it's also the first one after testnocgo):
https://cs.opensource.google/go/go/+/master:src/cmd/dist/test.go;l=1228;drc=d7fcb5cf80953f1d63246f1ae9defa60c5ce2d76

That would be -buildmode=static -ldflags='-linkmode=external -extldflags "-static -pthread"', I think?

Or go tool dist test cmd/cgo/internal/test:static.

Ah, which I see is what you already tried. 😞

As it magically passes now, we still want to investigate it but it is not currently a release blocker.