runtime: "fatal: systemstack called from unexpected goroutine" on armv6 #38539

diamondo25 · 2020-04-20T14:13:51Z

Note: I am using karalabe's xgo for compiling to ARM6

What version of Go are you using (`go version`)?

$ go version
go version go1.12.17 linux/amd64

Does this issue reproduce with the latest release?

Yes, 1.12.17 and 1.14.0 both seem affected.

What did you do?

Cross-compile application with github.com/mattn/go-sqlite3 as only import to armv6, with CGO_ENABLED=1

Code:

package main

import _ "github.com/mattn/go-sqlite3"

func main() {
	println("Hello")
}

Longer story is that my host OS is macOS, so I am using karalabe/go-1.12.12 by default, as target. However, I have also tested this on go-1.12.17 (cd docker/go-1.12.17 && docker build -t karalabe/xgo-1.12.17) and the latest, 1.14.0 (cd docker/go-1.14.0 && docker build -t karalabe/xgo-1.14.0), but no difference.

I am able to get Segfault on three occasions:

use code without gomod, output of built executable will be Segmentation fault
use code with gomod (only go get github.com/mattn/go-sqlite3 as package), output of executable will be Segmentation fault
use code with gomod (only go get github.com/mattn/go-sqlite3@v1.11.0 as package), output of executable will be fatal: systemstack called from unexpected goroutineSegmentation fault

Building

Test environment

Raspberry Pi Zero (not W). Building through cross-compilation because of no internet.

pi@raspberrypi:~ $ cat /proc/cpuinfo
processor	: 0
model name	: ARMv6-compatible processor rev 7 (v6l)
BogoMIPS	: 997.08
Features	: half thumb fastmult vfp edsp java tls
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xb76
CPU revision	: 7

Hardware	: BCM2835
Revision	: 900093
Serial		: 0000000088a33a5e

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.14.98+ #1200 Tue Feb 12 20:11:02 GMT 2019 armv6l GNU/Linux

Not using gomod

Build code:

$ mkdir diamondo25 && cd diamondo25
# write the main.go file from the code above
$ nano main.go
$ docker run -it -v $PWD:/code/ -w /code/ --entrypoint bash karalabe/xgo-1.12.17  
root@91db62d6dde7:/code# export CC=arm-linux-gnueabihf-gcc-5 GOOS=linux GOARCH=arm GOARM=6 CGO_ENABLED=1

root@91db62d6dde7:/code# CGO_CFLAGS='-march=armv6' CGO_CXXFLAGS='-march=armv6' go install std

root@91db62d6dde7:/code# go get github.com/mattn/go-sqlite3

root@91db62d6dde7:/code# go build -o hello .

root@91db62d6dde7:/code# exit

There's now a hello executable in the 'diamondo25' folder.

Copy the binary to a armv6l box. Running the executable SEGFAULTs right away:

pi@raspberrypi:~ $ ./hello
Segmentation fault

pi@raspberrypi:~ $ file hello
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=c60aae380bfbd3f2292bfa0a5718ecaece5cb328, not stripped

GDB doesn't help much:

pi@raspberrypi:~ $ gdb hello
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hello...done.
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/pi/hello.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) r
Starting program: /home/pi/hello
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
runtime.tracebackHexdump (stk=..., frame=0xbefffeb8, bad=3204447949) at /usr/local/go/src/runtime/traceback.go:992
992	/usr/local/go/src/runtime/traceback.go: No such file or directory.
(gdb) bt
#0  runtime.tracebackHexdump (stk=..., frame=0xbefffeb8, bad=3204447949) at /usr/local/go/src/runtime/traceback.go:992
#1  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Using gomod

$ mkdir diamondo25_gomod && cd diamondo25_gomod
# write the main.go file from the code above
$ nano main.go
$ docker run -it -v $PWD:/code/ -w /code/ --entrypoint bash karalabe/xgo-1.12.17
root@e35c28fde210:/code# go mod init hello
go: creating new go.mod: module hello

root@e35c28fde210:/code# go get github.com/mattn/go-sqlite3 
go: finding github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: downloading github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: extracting github.com/mattn/go-sqlite3 v2.0.3+incompatible

root@e35c28fde210:/code# export CC=arm-linux-gnueabihf-gcc-5 GOOS=linux GOARCH=arm GOARM=6 CGO_ENABLED=1

root@e35c28fde210:/code# CGO_CFLAGS='-march=armv6' CGO_CXXFLAGS='-march=armv6' go install std

root@e35c28fde210:/code# go build -o hello .
# Keep open for next variant

There's now a hello executable in the 'diamondo25_gomod' folder. The building of this executable took longer than the previous, no-gomod variant.

Running this also reports Segmentation fault

Using gomod with go-sqlite3@v1.11.0

Using the docker container shell, change the go-sqlite3 module to v1.11.0

root@e35c28fde210:/code# go get github.com/mattn/go-sqlite3@v1.11.0
go: finding github.com/mattn/go-sqlite3 v1.11.0
go: downloading github.com/mattn/go-sqlite3 v1.11.0
go: extracting github.com/mattn/go-sqlite3 v1.11.0

root@c1619b650cc7:/code# go build -o hello .

Now, copying and running this executable will throw the fatal: systemstack called from unexpected goroutineSegmentation fault error:

pi@raspberrypi:~ $ ./hello
fatal: systemstack called from unexpected goroutineSegmentation fault

GDB reports different info this time:

pi@raspberrypi:~ $ gdb hello
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hello...
done.
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/pi/hello.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) r
Starting program: /home/pi/hello
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
fatal: systemstack called from unexpected goroutine
Program received signal SIGSEGV, Segmentation fault.
runtime.abort () at /usr/local/go/src/runtime/asm_arm.s:801
801	/usr/local/go/src/runtime/asm_arm.s: No such file or directory.
(gdb) bt
#0  runtime.abort () at /usr/local/go/src/runtime/asm_arm.s:801
#1  0x00063a48 in runtime.badsystemstack () at /usr/local/go/src/runtime/stubs.go:62
#2  0x00049748 in runtime.throw (s=...) at /usr/local/go/src/runtime/panic.go:610
#3  0x00000012 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

It is unclear where it really crashes, as it never reports the stack. I think this is causing the fatal exception.

The text was updated successfully, but these errors were encountered:

diamondo25 · 2020-04-20T14:19:54Z

Buildlog for v1.11 library

root@c1619b650cc7:/code# go clean -cache -modcache  -r
root@c1619b650cc7:/code# go get github.com/mattn/go-sqlite3@v1.11.0
go: finding github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: finding github.com/mattn/go-sqlite3 v1.11.0
go: downloading github.com/mattn/go-sqlite3 v1.11.0
go: extracting github.com/mattn/go-sqlite3 v1.11.0
root@c1619b650cc7:/code# go clean -cache -modcache  -r
root@c1619b650cc7:/code# go build -v -x -o hello . &> buildlog_v1.11.txt

buildlog_v1.11.txt

diamondo25 · 2020-04-20T14:23:08Z

Buildlog with latest (v2.0.3+incompatible) library

root@c1619b650cc7:/code# go clean -cache -modcache  -r
root@c1619b650cc7:/code# go get github.com/mattn/go-sqlite3        
go: finding github.com/mattn/go-sqlite3 v1.11.0
go: finding github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: downloading github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: extracting github.com/mattn/go-sqlite3 v2.0.3+incompatible
root@c1619b650cc7:/code# go clean -cache -modcache  -r
root@c1619b650cc7:/code# go build -v -x -o hello . &> buildlog_v2.0.3.txt

buildlog_v2.0.3.txt

andybons · 2020-04-20T15:33:05Z

@aclements

diamondo25 · 2020-04-21T12:11:33Z

I've got myself the cables to download the packages etc. Now installing the sqlite package, so see you next year lol.

(this really takes time, as its compiling on a single core)

diamondo25 · 2020-04-21T12:36:41Z

When I built directly on the pi, it works.
buildlog_v2.0.3.txt

total 9400
   4 drwxr-xr-x 2 pi pi    4096 Apr 21 13:33 .
   4 drwxr-xr-x 8 pi pi    4096 Apr 21 12:56 ..
   4 -rw-r--r-- 1 pi pi      28 Apr 21 13:34 buildlog_v2.0.3.txt
4692 -rwxr-xr-x 1 pi pi 4800652 Apr 21 13:34 hello
   4 -rw-r--r-- 1 pi pi      89 Apr 21 12:45 main.go
pi@raspberrypi:~/diamondo25 $ go env
GO111MODULE=""
GOARCH="arm"
GOBIN=""
GOCACHE="/home/pi/.cache/go-build"
GOENV="/home/pi/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/pi/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm"
GCCGO="gccgo"
GOARM="6"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build150874878=/tmp/go-build -gno-record-gcc-switches"


pi@raspberrypi:~/diamondo25 $ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/6/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Raspbian 6.3.0-18+rpi1+deb9u1' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=arm-linux-gnueabihf- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-armhf/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-armhf --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-armhf --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 6.3.0 20170516 (Raspbian 6.3.0-18+rpi1+deb9u1)

diamondo25 · 2020-04-21T12:45:27Z

The xgo docker image runs the following version of gcc:

root@4b1de367ed35:/code# arm-linux-gnueabihf-gcc-5 -v
Using built-in specs.
COLLECT_GCC=arm-linux-gnueabihf-gcc-5
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/arm-linux-gnueabihf/5/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-armhf-cross/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-armhf-cross --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-armhf-cross --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libgcj --enable-objc-gc --enable-multiarch --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --disable-werror --enable-multilib --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=arm-linux-gnueabihf --program-prefix=arm-linux-gnueabihf- --includedir=/usr/arm-linux-gnueabihf/include
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4)

andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 20, 2020

andybons added this to the Unplanned milestone Apr 20, 2020

gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: "fatal: systemstack called from unexpected goroutine" on armv6 #38539

runtime: "fatal: systemstack called from unexpected goroutine" on armv6 #38539

diamondo25 commented Apr 20, 2020 •

edited

diamondo25 commented Apr 20, 2020 •

edited

diamondo25 commented Apr 20, 2020 •

edited

andybons commented Apr 20, 2020

diamondo25 commented Apr 21, 2020

diamondo25 commented Apr 21, 2020

diamondo25 commented Apr 21, 2020

runtime: "fatal: systemstack called from unexpected goroutine" on armv6 #38539

runtime: "fatal: systemstack called from unexpected goroutine" on armv6 #38539

Comments

diamondo25 commented Apr 20, 2020 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What did you do?

Building

Test environment

Not using gomod

Using gomod

Using gomod with go-sqlite3@v1.11.0

diamondo25 commented Apr 20, 2020 • edited

diamondo25 commented Apr 20, 2020 • edited

andybons commented Apr 20, 2020

diamondo25 commented Apr 21, 2020

diamondo25 commented Apr 21, 2020

diamondo25 commented Apr 21, 2020

diamondo25 commented Apr 20, 2020 •

edited

What version of Go are you using (`go version`)?

diamondo25 commented Apr 20, 2020 •

edited

diamondo25 commented Apr 20, 2020 •

edited