Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: Crash in GC on Debian riscv64 (qemu) #54104

Closed
phpeterson-usf opened this issue Jul 28, 2022 · 9 comments
Closed

runtime: Crash in GC on Debian riscv64 (qemu) #54104

phpeterson-usf opened this issue Jul 28, 2022 · 9 comments
Assignees
@mknyszek
Labels
arch-riscv Issues solely affecting the riscv64 architecture. compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@phpeterson-usf
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.18.4 darwin/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

I am cross-compiling the micro editor on my Mac targeting a Debian riscv64 guest under QEMU. The Debian guest is Linux debian 5.18.0-2-riscv64 https://github.com/zyedidia/micro/issues/1 SMP Debian 5.18.5-1 (2022-06-16) riscv64 GNU/Linux

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/phil/Library/Caches/go-build"
GOENV="/Users/phil/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/phil/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/phil/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.18.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/phil/riscv64-ubuntu/micro/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bl/ncrkxg1d6r130_5pb8mn3y3m0000gn/T/go-build3611809763=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

  1. After cloning the repo, the build command is GOOS=linux GOARCH=riscv64 CGO_ENABLED=0 make build.
  2. The resulting binary is called micro and I scp that to the Debian guest

What did you expect to see?

I expect to run the micro editor

What did you see instead?

The micro editor crashes on launch with the output below, apparently showing a crash in the Go GC. I opened an issue in the micro repo, but the author suggests that the root cause is inside Go.

Interestingly, the same micro binary runs on an Ubuntu guest Linux ubuntu 5.13.0-1026-generic #29~20.04.1-Ubuntu SMP Fri Jun 3 11:55:52 UTC 2022 riscv64 riscv64 riscv64 GNU/Linux but crashes on launch on the Debian guest.

stack trace stack trace
runtime: lfstack.push invalid packing: node=0xffffff8cc2b980 cnt=0x1 packed=0xffff8cc2b9800001 -> node=0xffff8cc2b980

fatal error: lfstack.push


runtime stack:

runtime.throw({0x5966ba, 0xc})

runtime/panic.go:992 +0x58

runtime.(*lfstack).push(0xaf62e8, 0xffffff8cc2b980)

runtime/lfstack.go:30 +0x144

runtime.(*spanSetBlockAlloc).free(...)

runtime/mspanset.go:292

runtime.(*spanSet).reset(0xaef898)

runtime/mspanset.go:265 +0xac

runtime.finishsweep_m()

runtime/mgcsweep.go:260 +0xc8

runtime.gcStart.func1()

runtime/mgc.go:664 +0x1c

runtime.systemstack()

runtime/asm_riscv64.s:133 +0x50


goroutine 1 [running]:

runtime.systemstack_switch()

runtime/asm_riscv64.s:96 +0x8 fp=0xc0004b3720 sp=0xc0004b3718 pc=0x74098

runtime.gcStart({0x0, 0x0, 0x0})

runtime/mgc.go:663 +0x500 fp=0xc0004b3798 sp=0xc0004b3720 pc=0x29188

runtime.mallocgc(0x30, 0x5576e0, 0x1)

runtime/malloc.go:1205 +0x71c fp=0xc0004b3810 sp=0xc0004b3798 pc=0x1bdb4

runtime.growslice(0x5576e0, {0x0, 0x0, 0x0}, 0x1)

runtime/slice.go:278 +0x4e0 fp=0xc0004b3860 sp=0xc0004b3810 pc=0x5e4b8

regexp/syntax.(*compiler).inst(...)

regexp/syntax/compile.go:164

regexp/syntax.(*compiler).init(...)

regexp/syntax/compile.go:83

regexp/syntax.Compile(0xc000131810)

regexp/syntax/compile.go:73 +0x9c fp=0xc0004b3920 sp=0xc0004b3860 pc=0x151214

regexp.compile({0x59c8fd, 0x14}, 0xd4, 0x0)

regexp/regexp.go:180 +0xa4 fp=0xc0004b39b8 sp=0xc0004b3920 pc=0x167c94

regexp.Compile(...)

regexp/regexp.go:135

regexp.MustCompile({0x59c8fd, 0x14})

regexp/regexp.go:315 +0x3c fp=0xc0004b3a38 sp=0xc0004b39b8 pc=0x1688ac

main.LoadInput({0xc00001c310, 0x0, 0x0})

github.com/zyedidia/micro/v2/cmd/micro/micro.go:166 +0x14c fp=0xc0004b3bb8 sp=0xc0004b3a38 pc=0x4af5d4

main.main()

github.com/zyedidia/micro/v2/cmd/micro/micro.go:319 +0x748 fp=0xc0004b3f80 sp=0xc0004b3bb8 pc=0x4b0988

runtime.main()

runtime/proc.go:250 +0x228 fp=0xc0004b3fd8 sp=0xc0004b3f80 pc=0x48480

runtime.goexit()

runtime/asm_riscv64.s:497 +0x4 fp=0xc0004b3fd8 sp=0xc0004b3fd8 pc=0x75ec4


goroutine 6 [sleep]:

time.Sleep(0x1dcd65000)

runtime/time.go:194 +0x150

github.com/zyedidia/micro/v2/internal/buffer.backupThread()

github.com/zyedidia/micro/v2/internal/buffer/backup.go:37 +0x28

created by github.com/zyedidia/micro/v2/internal/buffer.init.0

github.com/zyedidia/micro/v2/internal/buffer/backup.go:52 +0x68


goroutine 8 [syscall]:

os/signal.signal_recv()

runtime/sigqueue.go:151 +0x38

os/signal.loop()

os/signal/signal_unix.go:23 +0x1c

created by os/signal.Notify.func1.1

os/signal/signal.go:151 +0x2c


goroutine 9 [select]:

github.com/zyedidia/tcell/v2.(*tScreen).mainLoop(0xc000190000)

github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:1580 +0xe0

created by github.com/zyedidia/tcell/v2.(*tScreen).Init

github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:210 +0x6cc


goroutine 10 [IO wait]:

internal/poll.runtime_pollWait(0xffffff8cc35198, 0x72)

runtime/netpoll.go:302 +0xe0

internal/poll.(*pollDesc).wait(0xc000058e58, 0x72, 0x1)

internal/poll/fd_poll_runtime.go:83 +0x34

internal/poll.(*pollDesc).waitRead(...)

internal/poll/fd_poll_runtime.go:88

internal/poll.(*FD).Read(0xc000058e40, {0xc0003fb000, 0x1000, 0x1000})

internal/poll/fd_unix.go:167 +0x1f0

os.(*File).read(...)

os/file_posix.go:31

os.(*File).Read(0xc00000c780, {0xc0003fb000, 0x1000, 0x1000})

os/file.go:119 +0x70

github.com/zyedidia/tcell/v2.(*tScreen).inputLoop(0xc000190000)

github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:1633 +0x94

created by github.com/zyedidia/tcell/v2.(*tScreen).Init

github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:211 +0x724
@phpeterson-usf phpeterson-usf mentioned this issue Jul 28, 2022
Closed
@mengzhuo
Copy link
Contributor

It will be fixed after this CL been merged.
https://go-review.googlesource.com/c/go/+/409055

@mengzhuo mengzhuo changed the title affected/package: Crash in GC on Debian riscv64 runtime: Crash in GC on Debian riscv64 (qemu) Jul 28, 2022
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 28, 2022
@mengzhuo mengzhuo added arch-riscv Issues solely affecting the riscv64 architecture. compiler/runtime Issues related to the Go compiler and/or runtime. and removed compiler/runtime Issues related to the Go compiler and/or runtime. labels Jul 28, 2022
@phpeterson-usf
Copy link
Author

Thanks. I would be thrilled if this could make the 1.19 release train to support my teaching objectives.

@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 28, 2022
@cherrymui cherrymui added this to the Backlog milestone Jul 28, 2022
@cherrymui
Copy link
Member

Can this reproduce on a real hardware?

For QEMU, I guess it would be possible to configure it to not enable SV57 mode?

@phpeterson-usf
Copy link
Author

phpeterson-usf commented Jul 28, 2022 •
edited

I believe SV57 is enabled in the 5.18 kernel which is distributed with Debian/riscv64. I have looked unsuccessfully for runtime configuration for that feature. If there's a way to change that in QEMU, I don't know what it is.

I don't have an Unmatched board, and SiFive discontinued them in favor of a pre-release collaboration with Intel.

If it's too late for 1.19, the two possible workarounds I see are:

  1. Downgrade the kernel to pre-SV57
  2. Build Go from source, patching in Ian's change myself

@phpeterson-usf
Copy link
Author

FWIW, I built from go1.19rc2 + Ian's change (e9f157e.diff) and cross-compiled micro to avoid bootstrapping Go on riscv64. The resulting micro binary seems to work fine on my Debian/riscv64 guest.

@mengzhuo
Copy link
Contributor

mengzhuo commented Jul 29, 2022 •
edited

Can this reproduce on a real hardware?

For QEMU, I guess it would be possible to configure it to not enable SV57 mode?

Most of existed real hardwares (that I can get) using sv39

  • Sifive unmatched(Go build farm): sv39
  • Starfive v1: sv39
  • Allwinner D1: sv39

Qemu configured with -cpu=sifive-u54 to disable SV57.

@yuzibo
Copy link

yuzibo commented Jul 29, 2022

Hi,
I just have Unmatched boards by hand and I tested it easily:

vimer@unmatched:~/go/proj/bin$ uname -a
Linux unmatched 5.18.0-2-riscv64 #1 SMP Debian 5.18.5-1 (2022-06-16) riscv64 GNU/Linux
vimer@unmatched:~/go/proj/bin$ go env
GO111MODULE=""
GOARCH="riscv64"
GOBIN=""
GOCACHE="/home/vimer/.cache/go-build"
GOENV="/home/vimer/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="riscv64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/vimer/go/proj/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/vimer/go/proj"
GOPRIVATE=""
GOPROXY="https://proxy.golang.com.cn"
GOROOT="/home/vimer/go/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/vimer/go/go/pkg/tool/linux_riscv64"
GOVCS=""
GOVERSION="go1.18.4"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2096220389=/tmp/go-build -gno-record-gcc-switches"

vimer@unmatched:~/go/proj/bin$ ./micro --version
Version: 2.0.11-dev.100
Commit hash: 225927b9
Compiled on July 29, 2022

vimer@unmatched:~/go/proj/bin$ ip a | ./micro

The micro editor runs fine in my view also.
Please let me know if there are any issues.

@gopherbot
Copy link

Change https://go.dev/cl/409055 mentions this issue: runtime: support riscv64 SV57 mode

@mknyszek mknyszek self-assigned this Aug 3, 2022
gopherbot pushed a commit that referenced this issue Aug 4, 2022
@dvyukov
runtime: support riscv64 SV57 mode
1e3c19f 
riscv64 has SV57 mode when user-space VA is 56 bits.
Linux kernel recently got support for this mode and Go binaries started crashing as:

runtime: lfstack.push invalid packing: node=0xffffff5908a940 cnt=0x1
packed=0xffff5908a9400001 -> node=0xffff5908a940

Adjust lfstack code to use only 8 top bits of pointers on riscv64.

For context see:
https://groups.google.com/g/syzkaller-bugs/c/lU0GQTZoNQQ/m/O_c3vmE3AAAJ

Update #54104

Change-Id: Ib5d3d6a79c0c6eddf11618d73fcc8bc1832a9c25
Reviewed-on: https://go-review.googlesource.com/c/go/+/409055
Reviewed-by: Joel Sing <joel@sing.id.au>
Reviewed-by: Meng Zhuo <mzh@golangcn.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
jproberts pushed a commit to jproberts/go that referenced this issue Aug 10, 2022
@dvyukov @jproberts
runtime: support riscv64 SV57 mode
f0708a0 
riscv64 has SV57 mode when user-space VA is 56 bits.
Linux kernel recently got support for this mode and Go binaries started crashing as:

runtime: lfstack.push invalid packing: node=0xffffff5908a940 cnt=0x1
packed=0xffff5908a9400001 -> node=0xffff5908a940

Adjust lfstack code to use only 8 top bits of pointers on riscv64.

For context see:
https://groups.google.com/g/syzkaller-bugs/c/lU0GQTZoNQQ/m/O_c3vmE3AAAJ

Update golang#54104

Change-Id: Ib5d3d6a79c0c6eddf11618d73fcc8bc1832a9c25
Reviewed-on: https://go-review.googlesource.com/c/go/+/409055
Reviewed-by: Joel Sing <joel@sing.id.au>
Reviewed-by: Meng Zhuo <mzh@golangcn.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
@4a6f656c
Copy link
Contributor

This should be fixed via https://go.dev/cl/409055, closing.

@MaxTheMooshroom MaxTheMooshroom mentioned this issue Jul 6, 2023
Closed
@golang golang locked and limited conversation to collaborators Sep 17, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@mknyszek mknyszek

Labels
arch-riscv Issues solely affecting the riscv64 architecture. compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
7 participants
@mengzhuo @mknyszek @yuzibo @4a6f656c @gopherbot @cherrymui @phpeterson-usf