Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: SIGILL on ARMv7 after a C++ throw #20089

Closed
bassam opened this issue Apr 23, 2017 · 11 comments
Closed

cmd/compile: SIGILL on ARMv7 after a C++ throw #20089

bassam opened this issue Apr 23, 2017 · 11 comments

Comments

@bassam
Copy link

bassam commented Apr 23, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go 1.8.1

What operating system and processor architecture are you using (go env)?

GOARCH="arm"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/rook/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
GOARM="7"
CC="arm-linux-gnueabihf-gcc"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build245517363=/tmp/go-build -gno-record-gcc-switches"
CXX="arm-linux-gnueabihf-g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-I/home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-L/home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/lib"

What did you do?

I cross compiled Rook for armhf (see rook/rook#497) but it seems to fail with a SIGILL on an Armv7 (Scaleway C1 machines). Everything works fine for aarch64 and amd64 but not arm.

Leading up to the SIGILL I see a SIGSEGV

Thread 11 "rookd" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 1814]
0x01d539c0 in __cxa_throw ()
(gdb) bt
#0  0x01d539c0 in __cxa_throw ()
#1  0x00ecfbb8 in boost::throw_exception<boost::bad_lexical_cast> (e=...)
    at /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/throw_exception.hpp:69
#2  0x016565b2 in boost::conversion::detail::throw_bad_cast<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long long> ()
    at /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/lexical_cast/bad_lexical_cast.hpp:92
#3  boost::lexical_cast<unsigned long long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (arg=...)
    at /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/lexical_cast.hpp:42
#4  validate (value=0x8213179c, error_message=0x8213181c)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config_validators.cc:34
#5  0x0104d57a in std::function<int (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const (__args#1=<optimized out>,
    __args#0=<optimized out>, this=0x791a7ec) at /usr/arm-linux-gnueabihf/include/c++/6/functional:2127
#6  md_config_t::set_val_impl (this=this@entry=0x78fc000, val=..., opt=opt@entry=0x791a7d8, error_message=error_message@entry=0x8213181c)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc:1059
#7  0x0104f5f2 in md_config_t::validate_default_settings (this=this@entry=0x78fc000)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc:1380
#8  0x01052d74 in md_config_t::md_config_t (this=0x78fc000)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc:185
#9  0x01072c82 in CephContext::CephContext (this=0x78f8000, module_type_=1, init_flags_=0)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/ceph_context.cc:501
#10 0x01079c00 in common_preinit (iparams=..., code_env=code_env@entry=CODE_ENVIRONMENT_LIBRARY, flags=flags@entry=0,
    data_dir_option=data_dir_option@entry=0x0)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/common_init.cc:46
#11 0x00f95052 in cephd_generate_secret_key (buf=0x97077180 "", len=128)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/libcephd/libcephd.cc:84
#12 0x00e4d014 in _cgo_f53ec84379ef_Cfunc_cephd_generate_secret_key ()
#13 0x000e1d88 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_arm.s:543
#14 0x00000000 in ?? ()

when I continue I see the following:

Thread 11 "rookd" received signal SIGILL, Illegal instruction.
runtime.semrelease (addr=0x0) at /usr/local/go/src/runtime/sema.go:184
(gdb) x/i $pc
=> 0xcae40 <runtime.semrelease+380>:    ldr     pc, [sp], #32
(gdb) bt
#0  runtime.semrelease (addr=0x0) at /usr/local/go/src/runtime/sema.go:184
#1  0x00000000 in ?? ()

Here's the cpu info:

processor	: 0
model name	: ARMv7 Processor rev 2 (v7l)
BogoMIPS	: 50.00
Features	: half thumb fastmult vfp edsp thumbee vfpv3 tls idiva idivt vfpd32 lpae 
CPU implementer	: 0x56
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0x584
CPU revision	: 2

Note that I'm cross compiling rook with arm-linux-gnueabihf-gcc, here's the version info:

rook@moby:~/go/src/github.com/rook/rook$ arm-linux-gnueabihf-gcc -v
arm-linux-gnueabihf-gcc (Ubuntu/Linaro 6.3.0-12ubuntu2) 6.3.0 20170406
Using built-in specs.
COLLECT_GCC=arm-linux-gnueabihf-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/arm-linux-gnueabihf/6/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 6.3.0-12ubuntu2' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-armhf-cross/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-armhf-cross --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-armhf-cross --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libgcj --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --disable-werror --enable-multilib --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=arm-linux-gnueabihf --program-prefix=arm-linux-gnueabihf- --includedir=/usr/arm-linux-gnueabihf/include
Thread model: posix

Note the --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb defaults.

Also note that rookd is a statically compiled binary and libc/libc++ are also statically compiled in.

What did you expect to see?

No SIGILL

@cherrymui
Copy link
Member

--with-mode=thumb

Could you try to disable thumb? Apparently the instruction in semrelease is valid, but it can be invalid in thumb mode. Go doesn't use thumb mode. It seems when it throws in thumb mode, it doesn't switch to arm mode.

@bassam
Copy link
Author

bassam commented Apr 23, 2017

@cherrymui I tried compiling our c/c++ code using -marm and I still see the failure. However, libc and libstdc++ are also compiled in thumb mode and I didn't change those. I would have to recompile libc and libstdc++ to avoid using thumb completely, right?

Is it possible to disable -marm in golang and just go with thumb?

FWIW, I also tried compiling with -mthumb-interwork but that didn't help.

@cherrymui
Copy link
Member

I tried compiling our c/c++ code using -marm and I still see the failure.

Could you post the new stack trace on failure? The stack trace in the earlier post shows that it goes into thumb mode very early:

#11 0x00f95052 in cephd_generate_secret_key (buf=0x97077180 "", len=128)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/libcephd/libcephd.cc:84

(because the PC is not 4-byte aligned)

However, libc and libstdc++ are also compiled in thumb mode and I didn't change those. I would have to recompile libc and libstdc++ to avoid using thumb completely, right?

I suspect libc supports both arm and thumb, and the linker should choose the right one. Could you paste your build process with go build -x? Thanks.

Is it possible to disable -marm in golang and just go with thumb?

No. Go assembler doesn't know thumb at all. For this to work it would require adding an entire backend.

@bassam
Copy link
Author

bassam commented Apr 24, 2017

I compiled all our c/c++ packages with -marm, here's the build log with -x:

# github.com/rook/rook/cmd/rookd
HEADER = -H4 -T0x11000 -D0x0 -R0x10000
searching for runtime.a in $WORK/runtime.a
searching for runtime.a in /home/rook/go/pkg/linux_arm_netgo/runtime.a
searching for runtime.a in /usr/local/go/pkg/linux_arm_netgo/runtime.a
searching for math.a in $WORK/math.a
searching for math.a in /home/rook/go/pkg/linux_arm_netgo/math.a
searching for math.a in /usr/local/go/pkg/linux_arm_netgo/math.a
 0.00 deadcode
 0.17 pclntab=7463342 bytes, funcdata total 1021601 bytes
 0.20 dodata
 0.24 dwarf
 0.33 symsize = 0
 0.68 reloc
 0.80 asmb
 0.84 datblk
 0.86 sym
 0.86 elfsym
 0.89 symsize = 725440
 0.91 symsize = 726416
 0.99 header
 1.05 host link: "arm-linux-gnueabihf-g++" "-marm" "-gdwarf-2" "-o" "/tmp/go-build027260696/github.com/rook/rook/cmd/rookd/_obj/exe/a.out" "-static" "/tmp/go-link-133287218/go.o" "/tmp/go-link-133287218/000000.o" "/tmp/go-link-133287218/000001.o" "/tmp/go-link-133287218/000002.o" "/tmp/go-link-133287218/000003.o" "/tmp/go-link-133287218/000004.o" "-L/home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/lib" "-ltcmalloc_minimal" "-Wl,--whole-archive" "-lpthread" "-Wl,--no-whole-archive" "-lcephd" "-lboost_system" "-lboost_thread" "-lboost_iostreams" "-lboost_random" "-lblkid" "-lrocksdb" "-lcurl" "-lz" "-lsnappy" "-lzstd" "-lcryptopp" "-lssl" "-lcrypto" "-laio" "-luuid" "-lexpat" "-lm" "-ldl" "-lresolv" "-L/home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/lib" "-L/home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/lib" "-g" "-O2" "-lpthread" "-g" "-O2" "-no-pie" "-static"
/home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/lib/libcephd.a(ClassHandler.cc.o): In function `ClassHandler::_load_class(ClassHandler::ClassData*)':
/home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/osd/ClassHandler.cc:151: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-133287218/000004.o: In function `mygetgrouplist':
/usr/local/go/src/os/user/getgrouplist_unix.go:15: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-133287218/000004.o: In function `mygetgrgid_r':
/usr/local/go/src/os/user/lookup_unix.go:38: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-133287218/000004.o: In function `mygetgrnam_r':
/usr/local/go/src/os/user/lookup_unix.go:43: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/lib/libcephd.a(civetweb.c.o): In function `set_uid_option':
/home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/civetweb/src/civetweb.c:10481: warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-133287218/000004.o: In function `mygetpwnam_r':
/usr/local/go/src/os/user/lookup_unix.go:33: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-133287218/000004.o: In function `mygetpwuid_r':
/usr/local/go/src/os/user/lookup_unix.go:28: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/lib/libcephd.a(civetweb.c.o): In function `mg_inet_pton':
/home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/civetweb/src/civetweb.c:5652: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/lib/libcrypto.a(b_sock.o): In function `BIO_gethostbyname':
/home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/src/openssl/crypto/bio/b_sock.c:121: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
25.83 cpu time
357758 symbols
395804 liveness data

and here's a new back trace (looks 4 byte aligned):

[Switching to LWP 24485]
0x0243e918 in __cxa_throw ()
(gdb) bt
#0  0x0243e918 in __cxa_throw ()
#1  0x00f263d8 in boost::throw_exception<boost::bad_lexical_cast> (e=...)
    at /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/throw_exception.hpp:69
#2  0x01a24864 in boost::conversion::detail::throw_bad_cast<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long long> ()
    at /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/lexical_cast/bad_lexical_cast.hpp:92
#3  boost::lexical_cast<unsigned long long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (arg=...)
    at /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/lexical_cast.hpp:42
#4  validate (value=0x5c, error_message=0x40)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config_validators.cc:34
#5  0x0115b884 in std::function<int (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const (__args#1=<optimized out>, 
    __args#0=<optimized out>, this=0x800a7ec) at /usr/arm-linux-gnueabihf/include/c++/6/functional:2127
#6  md_config_t::set_val_impl (this=this@entry=0x7fec000, val=..., opt=opt@entry=0x800a7d8, error_message=0x82931824, 
    error_message@entry=0x8293181c) at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc:1059
#7  0x0115e988 in md_config_t::validate_default_settings (this=this@entry=0x7fec000)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc:1380
#8  0x01162784 in md_config_t::md_config_t (this=0x7fec000)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc:185
#9  0x011873a4 in CephContext::CephContext (this=0x7fe8000, module_type_=1, init_flags_=0)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/ceph_context.cc:501
#10 0x011918dc in common_preinit (iparams=..., code_env=code_env@entry=CODE_ENVIRONMENT_LIBRARY, flags=flags@entry=0, 
    data_dir_option=data_dir_option@entry=0x0)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/common_init.cc:46
#11 0x01049a28 in cephd_generate_secret_key (buf=0x970e7480 "", len=128)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/libcephd/libcephd.cc:84
#12 0x00e6edbc in _cgo_f53ec84379ef_Cfunc_cephd_generate_secret_key ()
#13 0x00103b30 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_arm.s:543
#14 0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

still happening:

Thread 10 "rookd" received signal SIGILL, Illegal instruction.
runtime.semrelease (addr=0x0) at /usr/local/go/src/runtime/sema.go:184
184	/usr/local/go/src/runtime/sema.go: No such file or directory.
(gdb) bt
#0  runtime.semrelease (addr=0x0) at /usr/local/go/src/runtime/sema.go:184
#1  0x00000000 in ?? ()

and here's the instruction:

(gdb) x/i $pc
=> 0xecbe8 <runtime.semrelease+380>:	ldr	pc, [sp], #32

@cherrymui
Copy link
Member

@bassam Thank you for the new stack trace. The PCs look like indeed 4-byte aligned. To make sure it is in arm mode, could you print the instruction and $cpsr in gdb when it throws? If it looks like in thumb mode, could you print the instructions for the call sites on each frame, to see when it switches to thumb?

In your code, whom do you expect to handle the throw? C++, Go, or nothing (terminate program)? I don't know why/how it goes to runtime.semrelease. If it is in thumb mode, it could just jump wildly though.

The build log looks like a -v log from the linker, instead of go build -x. What I'd like to find out is how the compiler is invoked. Could you paste the commands you invoked the Go and C++ compiler? If you build Go code with go build or go install, just add a -x flag.

Is there a way I could build your code and reproduce it? Thank you.

@bassam
Copy link
Author

bassam commented Apr 25, 2017

@cherrymui, here's the instruction and cpsr at the throw:

(gdb) x/i $pc
=> 0x243e918 <__cxa_throw+20>:  ldr     r3, [r0, #4]
(gdb) print $cpsr
$1 = 48

also here are the instructions at each call site:

(gdb) f 1
#1  0x00f263d8 in boost::throw_exception<boost::bad_lexical_cast> (e=...)
    at /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/throw_exception.hpp:69
69      /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/throw_exception.hpp: No such file or directory.
(gdb) x/i $pc
=> 0xf263d8 <boost::throw_exception<boost::bad_lexical_cast>(boost::bad_lexical_cast const&)+196>:      subeq   r10, r10, #204, 24      ; 0xcc00
(gdb) f 2
#2  0x01a24864 in boost::conversion::detail::throw_bad_cast<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long long> ()
    at /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/lexical_cast/bad_lexical_cast.hpp:92
92      /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/lexical_cast/bad_lexical_cast.hpp: No such file or directory.
(gdb) x/i $pc
=> 0x1a24864 <validate(md_config_t::option_rbd_default_features_t*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+228>: add     r2, sp, #88     ; 0x58
(gdb) f 3
#3  boost::lexical_cast<unsigned long long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (arg=...)
    at /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/lexical_cast.hpp:42
42      /home/rook/go/src/github.com/rook/rook/.work/build/arm-linux-gnueabihf/include/boost/lexical_cast.hpp: No such file or directory.
(gdb) x/i $pc
=> 0x1a24864 <validate(md_config_t::option_rbd_default_features_t*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+228>: add     r2, sp, #88     ; 0x58
(gdb) f 4
#4  validate (value=0x5c, error_message=0x40) at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config_validators.cc:34
34      /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config_validators.cc: No such file or directory.
(gdb) x/i $pc
=> 0x1a24864 <validate(md_config_t::option_rbd_default_features_t*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+228>: add     r2, sp, #88     ; 0x58
(gdb) f 5
#5  0x0115b884 in std::function<int (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const (__args#1=<optimized out>, __args#0=<optimized out>, this=0x800a7ec) at /usr/arm-linux-gnueabihf/include/c++/6/functional:2127
2127    /usr/arm-linux-gnueabihf/include/c++/6/functional: No such file or directory.
(gdb) x/i $pc
=> 0x115b884 <md_config_t::set_val_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, md_config_t::config_option const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+140>:    subs    r7, r0, #0
(gdb) f 6
#6  md_config_t::set_val_impl (this=this@entry=0x7fec000, val=..., opt=opt@entry=0x800a7d8, error_message=0x84d4f824, error_message@entry=0x84d4f81c)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc:1059
1059    /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc: No such file or directory.
(gdb) x/i $pc
=> 0x115b884 <md_config_t::set_val_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, md_config_t::config_option const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+140>:    subs    r7, r0, #0
(gdb) f 7
#7  0x0115e988 in md_config_t::validate_default_settings (this=this@entry=0x7fec000)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc:1380
1380    in /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc
(gdb) x/i $pc
=> 0x115e988 <md_config_t::validate_default_settings()+240>:    mov     r7, r0
(gdb) f 8
#8  0x01162784 in md_config_t::md_config_t (this=0x7fec000) at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc:185
185     in /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/config.cc
(gdb) x/i $pc
=> 0x1162784 <md_config_t::md_config_t()+15576>:        mov     r0, r8
(gdb) f 9
#9  0x011873a4 in CephContext::CephContext (this=0x7fe8000, module_type_=1, init_flags_=0)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/ceph_context.cc:501
501     /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/ceph_context.cc: No such file or directory.
(gdb) x/i $pc
=> 0x11873a4 <CephContext::CephContext(unsigned int, int)+92>:  add     r3, r4, #124    ; 0x7c
(gdb) f 10
#10 0x011918dc in common_preinit (iparams=..., code_env=code_env@entry=CODE_ENVIRONMENT_LIBRARY, flags=flags@entry=0, data_dir_option=data_dir_option@entry=0x0)
    at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/common_init.cc:46
46      /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/common/common_init.cc: No such file or directory.
(gdb) x/i $pc
=> 0x11918dc <common_preinit(CephInitParameters const&, code_environment_t, int, char const*)+72>:      mov     r1, r7
(gdb) f 11
#11 0x01049a28 in cephd_generate_secret_key (buf=0x9735f180 "", len=128) at /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/libcephd/libcephd.cc:84
84      /home/rook/go/src/github.com/rook/rook/external/src/ceph-kraken/src/libcephd/libcephd.cc: No such file or directory.
(gdb) x/i $pc
=> 0x1049a28 <cephd_generate_secret_key(char*, size_t)+80>:     mov     r4, r0
(gdb) f 12
#12 0x00e6edbc in _cgo_f53ec84379ef_Cfunc_cephd_generate_secret_key ()
(gdb) x/i $pc
=> 0xe6edbc <_cgo_f53ec84379ef_Cfunc_cephd_generate_secret_key+60>:     str     r0, [r11, #-8]
(gdb) f 13
#13 0x00103b30 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_arm.s:543
543     /usr/local/go/src/runtime/asm_arm.s: No such file or directory.
(gdb) x/i $pc
=> 0x103b30 <runtime.asmcgocall+88>:    mov     r5, r0

I expect the C++ code to handle the throw.

@bassam
Copy link
Author

bassam commented Apr 25, 2017

Also here's the go build -x output:

WORK=/tmp/go-build360064696
WORK=/tmp/go-build410289934
github.com/rook/rook
mkdir -p $WORK/github.com/rook/rook/_obj/
mkdir -p $WORK/github.com/rook/rook/_obj/exe/
cd /home/rook/go/src/github.com/rook/rook
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/github.com/rook/rook.a -trimpath $WORK -p main -complete -installsuffix nocgo -buildid fb8296665293dab984bc05b51c9072ca99f10c18 -D _/home/rook/go/src/github.com/rook/rook -I $WORK -I /home/rook/go/pkg/linux_arm_nocgo -pack ./main.go
cd .
/usr/local/go/pkg/tool/linux_amd64/link -o $WORK/github.com/rook/rook/_obj/exe/a.out -L $WORK -L /home/rook/go/pkg/linux_arm_nocgo -installsuffix nocgo -extld=arm-linux-gnueabihf-gcc -buildmode=exe -buildid=fb8296665293dab984bc05b51c9072ca99f10c18 -s -X github.com/rook/rook/pkg/version.Version=v0.3.3-58-gb214910-dirty $WORK/github.com/rook/rook.a
mkdir -p /home/rook/go/src/github.com/rook/rook/bin/linux_arm/
cp $WORK/github.com/rook/rook/_obj/exe/a.out /home/rook/go/src/github.com/rook/rook/bin/linux_arm/rook
github.com/rook/rook/cmd/rookd
mkdir -p $WORK/github.com/rook/rook/cmd/rookd/_obj/
mkdir -p $WORK/github.com/rook/rook/cmd/rookd/_obj/exe/
cd /home/rook/go/src/github.com/rook/rook/cmd/rookd
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/github.com/rook/rook/cmd/rookd.a -trimpath $WORK -p main -complete -installsuffix netgo -buildid c04c0521592a20996f7039de3203c3467a5668b2 -importmap github.com/bassam/cgosymbolizer=github.com/rook/rook/vendor/github.com/bassam/cgosymbolizer -importmap github.com/coreos/etcd/version=github.com/rook/rook/vendor/github.com/coreos/etcd/version -importmap github.com/coreos/pkg/capnslog=github.com/rook/rook/vendor/github.com/coreos/pkg/capnslog -importmap github.com/spf13/cobra=github.com/rook/rook/vendor/github.com/spf13/cobra -importmap k8s.io/client-go/kubernetes=github.com/rook/rook/vendor/k8s.io/client-go/kubernetes -importmap k8s.io/client-go/rest=github.com/rook/rook/vendor/k8s.io/client-go/rest -D _/home/rook/go/src/github.com/rook/rook/cmd/rookd -I $WORK -I /home/rook/go/pkg/linux_arm_netgo -pack ./api.go ./daemon.go ./main.go ./mds.go ./mon.go ./operator.go ./osd.go ./rgw.go ./tool.go ./version.go
cd .
/usr/local/go/pkg/tool/linux_amd64/link -o $WORK/github.com/rook/rook/cmd/rookd/_obj/exe/a.out -L $WORK -L /home/rook/go/pkg/linux_arm_netgo -installsuffix netgo -extld=arm-linux-gnueabihf-g++ -buildmode=exe -buildid=c04c0521592a20996f7039de3203c3467a5668b2 -X github.com/rook/rook/pkg/version.Version=v0.3.3-58-gb214910-dirty -extldflags -static $WORK/github.com/rook/rook/cmd/rookd.a
mkdir -p /home/rook/go/src/github.com/rook/rook/bin/linux_arm/
cp $WORK/github.com/rook/rook/cmd/rookd/_obj/exe/a.out /home/rook/go/src/github.com/rook/rook/bin/linux_arm/rookd

If you want to build it yourself here's the branch I'm using https://github.com/bassam/rook/tree/pr-armhf. Build instructions are here https://github.com/bassam/rook/tree/pr-armhf/build, I'm running the following to build for arm:

build/run make -j4 ALL_PLATFORMS=linux_arm release

I'm also happy to arrange an ssh session if you'd like to debug live. Thanks for all your help!

@cherrymui
Copy link
Member

@bassam From the CPSR it looks like it is in thumb mode. Unfortunately the PCs for each frame you pasted are the instruction after the call instruction. What I'd like to find out is whether it switches to thumb at one of calls (BLX instruction). If you print the previous instruction (or simply do disasm to disassemble a piece of code), it may be helpful. Is it convenient for you to share the binary? I could disassemble it to find out something, since you already pasted the PCs.

It is interesting that C++ doesn't handle the throw and it jumps to Go code. Probably messed up with arm/thumb mode... Do you have a standalone C++ binary that exercises the same code path? Maybe just call the function that you called with Cgo directly from C++ with the same arguments? If so, could you try to build it with -marm to see whether the throw is handled properly?

Thank you for the build instruction. I'll give it a try.

@bassam
Copy link
Author

bassam commented Apr 25, 2017

@cherrymui here are the binaries if you want to take a look:

https://www.dropbox.com/s/xtpiw19mmpof6mq/rook-v0.3.3-58-gb214910-dirty-linux-arm.tar.gz?dl=0
https://www.dropbox.com/s/qcbg9hw7v4yzafg/rook-v0.3.3-58-gb214910-dirty-linux-arm-debug.tar.gz?dl=0

The first link has the rookd binary and the second one has rookd.debug.

@cherrymui
Copy link
Member

@bassam Sorry for the delayed. I looked at your binary. It seems it goes to thumb mode at __cxa_throw, other frames are in ARM mode. This suggests that it may be related to libstdc++. Do you have a thumb-only libstdc++? If so, you'll probably need to install the ARM version of libstdc++.

If you have both ARM and thumb ones, it is useful to know how the linker is invoked. Do you use the Go linker or the C++ linker to produce the final binary? What flags are used? Thanks.

@bassam
Copy link
Author

bassam commented May 9, 2017

@cherrymui thanks that makes a lot of sense! I'll check the version of libstdc++ and hopefully that fixes the issue. I learned a lot about golang and ARM from this issue. Thanks again.

@bassam bassam closed this as completed May 9, 2017
@golang golang locked and limited conversation to collaborators May 9, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants