Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: crash on linux-ppc64le #59196

Closed
sumitd2 opened this issue Mar 23, 2023 · 10 comments
Closed

runtime: crash on linux-ppc64le #59196

sumitd2 opened this issue Mar 23, 2023 · 10 comments
Assignees
@laboger
Labels
arch-ppc64x compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-Linux
Milestone
Go1.21

Comments

@sumitd2
Copy link

sumitd2 commented Mar 23, 2023 •
edited

What version of Go are you using (go version)?

$ go version
1.20.2, master

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

Linux Ubuntu 22.04, ppc64le

go env Output
$ go env
GO111MODULE=""
GOARCH="ppc64le"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="ppc64le"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/sumit/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/sumit/go/pkg/tool/linux_ppc64le"
GOVCS=""
GOVERSION="devel go1.21-0aa14fca8c Wed Mar 22 21:17:50 2023 +0000"
GCCGO="gccgo"
GOPPC64="power8"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1361577659=/tmp/go-build -gno-record-gcc-switches"

What did you do?

SIGSEGV here: https://github.com/minio/minio/blob/0ff931dc765d5d8fdddcdf7687fec1b0b194fca9/cmd/bucket-replication-stats.go#L371

Ran following script on a fresh ppc64le Ubuntu 22.04 docker container (takes 15 min):

set -ex

CWD=$(pwd)

#Install linux  deps
if [ -f /etc/redhat-release ] ; then
	yum install wget git gcc make gcc-c++ curl -y
else
	apt-get update -y
	apt-get  install wget git build-essential curl -y
fi

#Install golang 1.20.1
wget https://go.dev/dl/go1.20.1.linux-ppc64le.tar.gz
rm -rf /usr/local/go
tar -C /usr/local -xzf go1.20.1.linux-ppc64le.tar.gz
rm -rf go1.20.1.linux-ppc64le.tar.gz
export PATH=$PATH:/usr/local/go/bin
go version

#Build and install golang/go master
git clone https://go.googlesource.com/go
cd go
git checkout master
cd src
./all.bash || true
PATH=$CWD/go/bin:$PATH
go version
cd ../../

#Install mc
curl -o /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-ppc64le/mc
chmod +x /usr/local/bin/mc

#Clone
git clone https://github.com/minio/minio.git
cd minio
git checkout RELEASE.2023-02-27T18-10-45Z

#Build
go mod tidy
make build
chmod +x minio

#Test
rm -rf /home/share
./minio server --address 127.0.0.1:9000 /home/share &
sleep 10
mc alias set myminio http://127.0.0.1:9000 minioadmin minioadmin
mc mb myminio/tns-bucket2
mc cp $(which mc) myminio/tns-bucket2
mc ls myminio/tns-bucket2
set +ex
echo "Please wait for 360 seconds for the crash..."
secs=$((6 * 60))
while [ $secs -gt 0 ]; do
   echo -ne "$secs\033[0K\r"
   sleep 1
   : $((secs--))
done

What did you expect to see?

No crash

What did you see instead?

unexpected fault address 0xffffffffffffefc8
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x3 addr=0xffffffffffffefc8 pc=0x1a3ec48]

goroutine 350 [running]:
runtime.throw({0x243009a?, 0x1a3e7b8?})
        runtime/panic.go:1076 +0x40 fp=0xc003f6a8d8 sp=0xc003f6a898 pc=0x4b050
runtime.sigpanic()
        runtime/signal_unix.go:851 +0x260 fp=0xc003f6a948 sp=0xc003f6a8d8 pc=0x64a50
github.com/minio/minio/cmd.(*ReplicationStats).calculateBucketReplicationStats(0xc002964690, {0xc006d50c80, 0xb}, {0x17f0000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...}, ...)
        github.com/minio/minio/cmd/bucket-replication-stats.go:371 +0x968 fp=0xc003f79b90 sp=0xc003f6a968 pc=0x1a3ec48
github.com/minio/minio/cmd.(*ReplicationStats).getAllLatest(0x0?, 0xc0070af4d0)
        github.com/minio/minio/cmd/bucket-replication-stats.go:282 +0x1f0 fp=0xc003f79e28 sp=0xc003f79b90 pc=0x1a3e0c0
github.com/minio/minio/cmd.(*ReplicationPool).saveStatsToDisk(0xc00054bce0)
        github.com/minio/minio/cmd/bucket-replication.go:3076 +0x258 fp=0xc003f79f98 sp=0xc003f79e28 pc=0x1a68108
github.com/minio/minio/cmd.NewReplicationPool.func5()
        github.com/minio/minio/cmd/bucket-replication.go:1612 +0x4c fp=0xc003f79fc0 sp=0xc003f79f98 pc=0x1a5bd3c
runtime.goexit()
        runtime/asm_ppc64x.s:900 +0x4 fp=0xc003f79fc0 sp=0xc003f79fc0 pc=0x85214
created by github.com/minio/minio/cmd.NewReplicationPool in goroutine 64
        github.com/minio/minio/cmd/bucket-replication.go:1612 +0x5e4
@sumitd2
Copy link
Author

sumitd2 commented Mar 23, 2023

cc: @Sapana-Khemkar @sethp @shixuguang

@laboger
Copy link
Contributor

laboger commented Mar 23, 2023

@sumitd2 Does this work on Go 1.20.1? I see that is installed first, then there is a build of Go master and that is used to build minio.
Is this being run on power9?

@sumitd2
Copy link
Author

sumitd2 commented Mar 23, 2023

@laboger 1.20.1 also crashes, its installed first because building go from source needs a go installation. And yes, on Power9.

@laboger
Copy link
Contributor

laboger commented Mar 23, 2023

Has this worked on a previous release?

@sumitd2
Copy link
Author

sumitd2 commented Mar 23, 2023 •
edited

@laboger go1.19 is crashing, go1.18.10 is working fine. Thats the narrowest I could get.

@heschi heschi changed the title cmd/go: Golang runtime crash on ppc64le runtime: crash on linux-ppc64le Mar 23, 2023
@heschi
Copy link
Contributor

heschi commented Mar 23, 2023

cc @golang/ppc64

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Mar 23, 2023
@heschi heschi added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 23, 2023
@heschi heschi added this to the Go1.21 milestone Mar 23, 2023
@gopherbot
Copy link

Change https://go.dev/cl/478920 mentions this issue: cmd/internal/obj/ppc64: fix incorrect base reg causing segv

@laboger laboger self-assigned this Mar 23, 2023
@sumitd2
Copy link
Author

sumitd2 commented Mar 24, 2023

@laboger I have verified, https://go.dev/cl/478920 fixes the issue on master.

@laboger
Copy link
Contributor

laboger commented Mar 24, 2023

@gopherbot please open the backport tracking issues for Go 1.20 and 1.19. This is an assembler regression that results in a serious error and has no workaround.

This was referenced Mar 24, 2023
Closed
Closed
@gopherbot
Copy link

Backport issue(s) opened: #59219 (for 1.19), #59220 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 24, 2023
@shixuguang shixuguang mentioned this issue Apr 4, 2023
Closed
@golang golang locked and limited conversation to collaborators Mar 23, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@laboger laboger

Labels
arch-ppc64x compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-Linux
Projects
None yet
Milestone
Go1.21
Development

No branches or pull requests
5 participants
@dmitshur @gopherbot @laboger @heschi @sumitd2