New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: "fatal error: runtime: name offset out of range" on Android 6.0.1 arm64-v8a #38838
Comments
Thanks for reporting. Is it possible to reproduce the problem with a smaller snippet? /cc @hyangah @hajimehoshi |
Possibly, but I'm neither a go programmer, nor do I actually have a device where the error is reproducible (edit: But I do have a tech savvy end user, who can execute on-device tests) However, we managed to confirm that this is a regression somewhere between
@ncw Do you have a clue where it fails? This would probably help to create a minimum reproducible snippet. We are executing go 1.13.4 (rclone 1.51.0, OK)2020/05/10 12:00:51 ERROR : Couldn't find home directory or read HOME or XDG_CONFIG_HOME environment variables. 2020/05/10 12:00:51 ERROR : Defaulting to storing config in current directory. 2020/05/10 12:00:51 ERROR : Use --config flag to workaround. 2020/05/10 12:00:51 ERROR : Error was: exec: "getent": executable file not found in $PATH rclone v1.51.0 - os/arch: android/arm64 - go version: go1.13.4 go 1.14.2 (rclone 1.51.0, nameOff)2020/05/10 12:02:43 %!s(fs.LogLevel=3 ): Couldn't find home directory or read HOME or XDG_CONFIG_HOME environment variables. 2020/05/10 12:02:43 %!s(fs.LogLevel=3 ): Defaulting to storing config in current directory. 2020/05/10 12:02:43 %!s(fs.LogLevel=3 ): Use --config flag to workaround. 2020/05/10 12:02:43 %!s(fs.LogLevel=3 ): Error was: exec: "getent": executable file not found in $PATH runtime: nameOff 0x57fb1cc0 out of range 0x5557e84a80 - 0x555831d640 fatal error: runtime: name offset out of range goroutine 1 [running, locked to thread]: runtime.throw(0x5557b5cb0e, 0x21) C:/sdev/Go/src/runtime/panic.go:1116 +0x4c fp=0x40000c99f0 sp=0x40000c99c0 pc=0x5556fad564 runtime.resolveNameOff(0x555807d280, 0x57fb1cc0, 0x5557e8d588) C:/sdev/Go/src/runtime/type.go:195 +0x290 fp=0x40000c9a60 sp=0x40000c99f0 pc=0x5556fd5148 reflect.resolveNameOff(0x555807d280, 0x5557fb1cc0, 0x5557e8d588) C:/sdev/Go/src/runtime/runtime1.go:482 +0x28 fp=0x40000c9a90 sp=0x40000c9a60 pc=0x5556fbeee0 reflect.(*rtype).nameOff(...) C:/sdev/Go/src/reflect/type.go:683 reflect.implements(0x555807d280, 0x5558029f20, 0x0) C:/sdev/Go/src/reflect/type.go:1520 +0x3bc fp=0x40000c9be0 sp=0x40000c9a90 pc=0x555700b1d4 reflect.(*rtype).Implements(0x5558029f20, 0x55582f1b20, 0x555807d280, 0x5557442300) C:/sdev/Go/src/reflect/type.go:1437 +0x60 fp=0x40000c9c10 sp=0x40000c9be0 pc=0x555700ac08 encoding/gob.implementsInterface(0x55582f1b20, 0x55581d83a0, 0x55582f1b20, 0x555807d280, 0x0) C:/sdev/Go/src/encoding/gob/type.go:142 +0x128 fp=0x40000c9c60 sp=0x40000c9c10 pc=0x5557442890 encoding/gob.validUserType(0x55582f1b20, 0x55581d83a0, 0x5558d2b680, 0x5556f86038, 0x400007f7c0) C:/sdev/Go/src/encoding/gob/type.go:90 +0x2f0 fp=0x40000c9cf0 sp=0x40000c9c60 pc=0x55574425a8 encoding/gob.userType(0x55582f1b20, 0x55581d83a0, 0x55581d83a0) C:/sdev/Go/src/encoding/gob/type.go:152 +0x28 fp=0x40000c9d40 sp=0x40000c9cf0 pc=0x5557442940 encoding/gob.mustGetTypeInfo(0x55582f1b20, 0x55581d83a0, 0x55582f1b20) C:/sdev/Go/src/encoding/gob/type.go:766 +0x28 fp=0x40000c9d90 sp=0x40000c9d40 pc=0x5557446030 encoding/gob.init() C:/sdev/Go/src/encoding/gob/type.go:259 +0x1bcc fp=0x40000c9e30 sp=0x40000c9d90 pc=0x5557449674 runtime.doInit(0x5558cc7c60) C:/sdev/Go/src/runtime/proc.go:5414 +0x94 fp=0x40000c9e70 sp=0x40000c9e30 pc=0x5556fbca3c runtime.doInit(0x5558cc6de0) C:/sdev/Go/src/runtime/proc.go:5409 +0x50 fp=0x40000c9eb0 sp=0x40000c9e70 pc=0x5556fbc9f8 runtime.doInit(0x5558cd2d00) C:/sdev/Go/src/runtime/proc.go:5409 +0x50 fp=0x40000c9ef0 sp=0x40000c9eb0 pc=0x5556fbc9f8 runtime.doInit(0x5558cd1c00) C:/sdev/Go/src/runtime/proc.go:5409 +0x50 fp=0x40000c9f30 sp=0x40000c9ef0 pc=0x5556fbc9f8 runtime.doInit(0x5558cc1540) C:/sdev/Go/src/runtime/proc.go:5409 +0x50 fp=0x40000c9f70 sp=0x40000c9f30 pc=0x5556fbc9f8 runtime.main() C:/sdev/Go/src/runtime/proc.go:190 +0x1d0 fp=0x40000c9fd0 sp=0x40000c9f70 pc=0x5556fb0348 runtime.goexit() C:/sdev/Go/src/runtime/asm_arm64.s:1148 +0x4 fp=0x40000c9fd0 sp=0x40000c9fd0 pc=0x5556fde03c |
All the traces have this in
So I would guess it is This suggests a fairly minimal go program to reproduce - can you try that? It should either do nothing or crash 🤞 package main
import (
"bytes"
"encoding/gob"
)
func main() {
var network bytes.Buffer
_ = gob.NewEncoder(&network)
} |
This happened in android/arm64 but I am not sure if this is a mobile-platform specific issue based on the stack trace (failure while retrieving runtime&reflect type info). Calling runtime experts for help. |
I don't see anything here that seems specific to the mobile platform. I don't know what is happening here. If someone has a reproduction that doesn't require any special setup or hardware, I imagine that this could be fixed easily. I agree that doing a git bisect would help. |
Unfortunately, the test program provided by @ncw does not trigger the bug. I also tried finding a QEMU image that reproduces the issue, but unfortunately, there does not seem to be an Android 6 arm64-v8a image, and images of newer versions don't show the behavior. I also tried to reduce rclone to only the bug-producing parts, but my go knowledge is not up to that task. Would it be possible to inject tons of print style debugging messages into rclone? That would be retrievable without special tooling by the user. |
:-(
But you've got hardware which can reproduce the problem?
We know just where it is going wrong, that is in the traceback so I don't think prints will help. The best thing to do now would be to git bisect the go compiler. So check out the go source. Compile it. Build an rclone with that compiler and test it. You then tell git whether that worked or not and git finds another commit to try. I've done it loads of times with the go source. It builds quickly so it doesn't take too long. I'm sure there are lots of tutorials out there to help with the git bisect process. There is even a script in the bin directory in the rclone source which can automate the process assuming you can automate the testing of the rclone binary on the phone The result will be the commit that broke the rclone compile and that will probably be enough for the team to figure out what is going on. |
No, otherwise I would have bisected it right away. I have to send a test package to an affected user, who then has to execute the package and report the results back. I did modify gvm in a way to mass-build your test from source with for every possible golang version, but until we have a test program that crashes, mass building would be a waste of resources. And mass-building rclone itself is not really feasible. However, I just had an idea and looked through old Google Play prelaunch reports - it does seem that Firebase has affected devices in their device farm. If it checks out, it might be possible to build an APK that can act as remote verifier. It will take a few days to build and test that. |
The prelaunch idea worked out. I used a Huawei P8 Lite from Firebase as proxy device since it shows the same behaviour - and test data from the affected user confirms it as well. The bisect result is this commit: d3c2b1f
Mentioning that in the official 1.14 changelog would have saved me a few days 🙄, especially since lld is still in testing as of the latest ndk release. Consequently, adding
We aren't 100% there, but with Firebase, I managed to confirm a working/breaking version in about 10 minutes/version. According to the play catalogue, filtered to Android 5.0-6.0.1 arm64-v8a, >1200 devices should be affected by this, so it is not all that unlikely that you actually have affected hardware somewhere in your drawer. Anyway, I can now confirm that go is broken when cross-compiling rclone |
awesome, @x0b! @eliasnaur @ianlancetaylor is it possible to revert the change (in particular, if ldd is still under testing)? |
I'm fine with reverting the change. I don't know anything about this. @x0b Thanks for the nice investigation of the problem. |
I suppose the change can be reverted, with the caveat that the Corellium virtual Androids may not have gold installed. I wonder why @steeve is not affected by similar crashes (or perhaps he is?). FWIW, ldd has been available since NDK r18b from late 2018, and "AOSP has switched to using LLD by default". Are you affected by android/ndk#843, @xob? |
We are not affected. Maybe it's because we are still running 1.13.8 + NDK 18b. We are also building with Bazel 2.0, which doesn't use |
Hang on, are you sure you're running |
You're right, more test data was needed. Regarding android/ndk#843 (default, I did those tests before reading the second comment):
Regarding linker parameter (
So the conclusion is...the NDK default is neither of those options. I guess the best thing would be to switch to explicit lld, which should continue to work when NDK switches to lld as default. |
I agree. Can you prepare a CL? |
I can try. I need some time to verify if any lld version is good and when to try to fall back to gold though. |
Change https://golang.org/cl/235017 mentions this issue: |
Change https://golang.org/cl/235098 mentions this issue: |
Change https://golang.org/cl/235157 mentions this issue: |
…rted CL 235017 is about to change the default Android linker to lld. lld doesn't support the --compress-debug-sections=zlib-gnu flag, but linkerFlagSupported doesn't take any alternative linkers specified with -fuse-ld into account. Updates #38838 Change-Id: I5f7422c06d40dedde2e4b070fc48398e8f822190 Reviewed-on: https://go-review.googlesource.com/c/go/+/235157 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Not sure
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
Cross-Compiled rclone for Android and ran it there inside of an app (original issue: x0b/rcx#58). About 2% of arm64-v8a devices seem to be affected, no complaints from the rest.
Build Commands/Env:
Run Env
Run command
What did you expect to see?
Normal program operation.
What did you see instead?
fatal error: runtime: name offset out of range
fatal error: runtime: name offset out of range
fatal error: runtime: name offset out of range
fatal error: fault
fatal error: runtime: name offset out of range
fatal error: runtime: name offset out of range
The text was updated successfully, but these errors were encountered: