What version of Go are you using (go version)?

go version go1.15 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ncw/.cache/go-build"
GOENV="/home/ncw/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ncw/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ncw/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/go/go1.15"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/go/go1.15/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build829175078=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I checked out the rclone repository: https://github.com/rclone/rclone - probably any large repo which builds under ARM will do.

I then ran this script

#!/bin/bash

echo "sequential build"
rm -rf ~/go/pkg/linux_arm
go clean -cache
GOOS=linux GOARCH=arm GOARM=6 go build -v -i -o sequential-i-v6.bin -trimpath 2>sequential-i-v6.log
GOOS=linux GOARCH=arm GOARM=7 go build -v -i -o sequential-i-v7.bin -trimpath 2>sequential-i-v7.log

echo "parallel build with -i"
rm -rf ~/go/pkg/linux_arm
go clean -cache
GOOS=linux GOARCH=arm GOARM=6 go build -v -i -o parallel-i-v6.bin -trimpath 2>parallel-i-v6.log &
GOOS=linux GOARCH=arm GOARM=7 go build -v -i -o parallel-i-v7.bin -trimpath 2>parallel-i-v7.log &

wait

echo "sequential build without -i"
rm -rf ~/go/pkg/linux_arm
go clean -cache
GOOS=linux GOARCH=arm GOARM=6 go build -v -o sequential-no-i-v6.bin -trimpath 2>sequential-no-i-v6.log
GOOS=linux GOARCH=arm GOARM=7 go build -v -o sequential-no-i-v7.bin -trimpath 2>sequential-no-i-v6.log

echo "parallel build without -i"
rm -rf ~/go/pkg/linux_arm
go clean -cache
GOOS=linux GOARCH=arm GOARM=6 go build -v -o parallel-no-i-v6.bin -trimpath 2>parallel-no-i-v6 &
GOOS=linux GOARCH=arm GOARM=7 go build -v -o parallel-no-i-v7.bin -trimpath 2>parallel-no-i-v7 &

wait

md5sum *.bin | sort

echo "done"

What did you expect to see?

I expected to see 8 binaries with 2 md5sums

What did you see instead?

I see that the binaries built in parallel with go build -i appear to be different

Note that the differing binaries sometimes differ from run to run.

run 1

1160905621e7ce72dd577d7cbbffc3b2  parallel-no-i-v7.bin
1160905621e7ce72dd577d7cbbffc3b2  sequential-i-v7.bin
1160905621e7ce72dd577d7cbbffc3b2  sequential-no-i-v7.bin
f2c7c50467ab0ec6354ec8f35805b65b  parallel-i-v7.bin

c048804419a985ba1f5bde7c7f2dd47f  parallel-no-i-v6.bin
c048804419a985ba1f5bde7c7f2dd47f  sequential-i-v6.bin
c048804419a985ba1f5bde7c7f2dd47f  sequential-no-i-v6.bin
126979dfe3987ad89a18af57f4089b02  parallel-i-v6.bin

run 2-3

Identical to above

run 4

1160905621e7ce72dd577d7cbbffc3b2  parallel-no-i-v7.bin
1160905621e7ce72dd577d7cbbffc3b2  sequential-i-v7.bin
1160905621e7ce72dd577d7cbbffc3b2  sequential-no-i-v7.bin
6a99ec2ac3e4ea4684882da699ac9d7b  parallel-i-v7.bin

c048804419a985ba1f5bde7c7f2dd47f  parallel-no-i-v6.bin
c048804419a985ba1f5bde7c7f2dd47f  sequential-i-v6.bin
c048804419a985ba1f5bde7c7f2dd47f  sequential-no-i-v6.bin
92e501e7ef370324003c0100ce49ea6f  parallel-i-v6.bin

Discussion

This came to my attention in rclone/rclone#4553 - ARMv6 builds for 1.53.0 result in "Illegal instruction" error.

The root cause of that issue seems to be rclone's builder building the ARMv6 and ARMv7 releases in parallel and somehow parts of the ARMv7 binary get mixed into the ARMv6 binary resulting in the illegal instruction on ARMv6. However the binary does run on ARMv7 indicating that it isn't just random corruption, it is probably compilation units that have got mixed up.

Rclone's builder is using go build -i for historical reasons (namely that is how you used to get build caching). Removing the -i from the command appears to fix the problem.