runtime: signal: illegal instruction on ios/arm64 #35851
Comments
dmitshur
added
mobile
|
Strangely, it failed at an earlier step on the next commit (https://build.golang.org/log/186ffe8d4a32f24c719cb49eddf6e37e73957239), but has been passing since then. The commit that fixed it doesn't seem related. |
|
@cherrymui Thanks. The darwin-arm64-corellium builder was passing for many commits between 01f15b6 and 67f0f83 (CL 208818). Do you know if that is it because this issue was sporadic? Do you think anything more needs to be done for this issue? |
|
Yeah, #35800 is sporadic, happens when a signal lands in some unlucky time. This also looks like a flake. I don't know what the cause is. |
|
2020-02-19T21:34:59-1e43298/darwin-arm64-corellium |
|
The workaround in #42774 will probably address this, but it relies on You ought to be able to use Apple FB8922558 aims to get this fixed in the kernel, because it is a kernel bug. |
|
I've upgraded all builders to iOS 14. I'm not sure what versions are covered by your "on newer OS versions", but I think Go should run on iOS < 14, at least until iOS 15 is out. |
|
Continuing the discussion from #42774 (comment) here.
@cherrymui I tried running ./all.bash while running the io benchmark concurrently. all.bash succeeded but the benchmark failed with an memory range error:
This issue being the iOS version of #41702 sounds plausible to me, since both crashes in this issue are from an exec (vet and compile). |
|
Change https://golang.org/cl/275293 mentions this issue: |
The iOS kernel has the same problem as the macOS kernel. Extend the workaround of #41702 (CL 262438 and CL 262817) to iOS. Updates #35851. Change-Id: I7ccec00dc96643c08c5be8b385394856d0fa0f64 Reviewed-on: https://go-review.googlesource.com/c/go/+/275293 Trust: Cherry Zhang <cherryyz@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
bcmills
changed the title
bcmills
changed the title
|
Yeah, apparently that was not. I'm planning to do more experiments. |
|
Change https://golang.org/cl/279489 mentions this issue: |
|
I tried to understand it better, but with the limited access on the builders I couldn't get anything useful. I'm inclined that this is a kernel bug or simulator bug (it is not a real iOS device, right?). It might be helpful to get a crash report or a core dump or something. |
Agreed, a kernel bug seems likely, especially given #42774. (I don't think we have any evidence to suggest a simulator bug at this point, but I can't rule it out either.) There was some discussion on #42774 (comment) of porting a workaround based on |
I recall I wrote a CL for the workaround and Elias tried but it wasn't helpful... |
|
Let me know if there are more experiments I can do. I believe |
|
I found 3 crash reports in the {"app_name":"compile","timestamp":"2021-06-21 07:14:05.00 -0700","app_version":"","slice_uuid":"466bad2f-2a67-3403-8bd6-112aecef5eac","build_version":"","platform":2,"share_with_app_devs":0,"is_first_party":1,"bug_type":"109","os_version":"iPhone OS 14.2 (18B92)","incident_id":"10DE5A2B-4502-438E-AF58-ED817E305251","name":"compile"} Incident Identifier: 10DE5A2B-4502-438E-AF58-ED817E305251 CrashReporter Key: f09d9091c97b4e4f6bb695146954c56a7f0495a7 Hardware Model: iPhone8,4 Process: compile [77475] Path: /private/var/tmp/workdir-host-ios-arm64-corellium-ios/go/pkg/tool/ios_arm64/compile Identifier: compile Version: ??? Code Type: ARM-64 (Native) Role: Unspecified Parent Process: go_bootstrap [77188] Coalition: org.golang.builder [157] {"app_name":"compile","timestamp":"2021-06-21 09:57:23.00 -0700","app_version":"","slice_uuid":"709e8d8b-fdc9-330c-bc07-60b0cd693652","build_version":"","platform":2,"share_with_app_devs":0,"is_first_party":1,"bug_type":"109","os_version":"iPhone OS 14.2 (18B92)","incident_id":"AA3C332D-EA5C-42CD-A6F0-56BF34DBC86F","name":"compile"} Incident Identifier: AA3C332D-EA5C-42CD-A6F0-56BF34DBC86F CrashReporter Key: f09d9091c97b4e4f6bb695146954c56a7f0495a7 Hardware Model: iPhone8,4 Process: compile [90439] Path: /private/var/tmp/workdir-host-ios-arm64-corellium-ios/go/pkg/tool/ios_arm64/compile Identifier: compile Version: ??? Code Type: ARM-64 (Native) Role: Unspecified Parent Process: go_bootstrap [90151] Coalition: org.golang.builder [157] {"app_name":"compile","timestamp":"2021-06-21 10:41:21.00 -0700","app_version":"","slice_uuid":"4e440b1b-ddbb-3cb9-9f4d-384ffadbf0bc","build_version":"","platform":2,"share_with_app_devs":0,"is_first_party":1,"bug_type":"109","os_version":"iPhone OS 14.2 (18B92)","incident_id":"8C91A684-E77A-4476-8930-BC84B04EEDB9","name":"compile"} Incident Identifier: 8C91A684-E77A-4476-8930-BC84B04EEDB9 CrashReporter Key: f09d9091c97b4e4f6bb695146954c56a7f0495a7 Hardware Model: iPhone8,4 Process: compile [91896] Path: /private/var/tmp/workdir-host-ios-arm64-corellium-ios/go/pkg/tool/ios_arm64/compile Identifier: compile Version: ??? Code Type: ARM-64 (Native) Role: Unspecified Parent Process: go_bootstrap [91607] Coalition: org.golang.builder [157] |
|
Thanks. The crash reports look a lot like the ones from #42774. Elias, do you confirm that CL https://go-review.googlesource.com/c/go/+/279489 doesn't help? Maybe there is other possibility for the kernel to fail to deliver signal... What is the builder setup? I think it is not a real iOS device, but some kind of VM? Does the SIGILL happen on real iOS devices? Thanks. |
|
I tried again to be sure: Crash report: |
|
Let me symbolize that for you (below). The crash is in The |
|
Thank you for your analysis, @markmentovai. I don't mind upgrading the builders to a newer iOS, but I don't want to paper over the issue for devices running older iOS versions. If the issue is the same as #42774, why don't our macOS workarounds work? |
|
@eliasnaur I left my thoughts on the
Spurious |
I'd still be interested in knowing why @cherrymui's CL doesn't work around the problem on 14.2 like a similar CL does on macOS. But I see your point now: a workaround based on sigaltstack will only cover [14.0;14.4] which is probably not worth the trouble. I'll upgrade the builders and report back. |
But that’s the builder and not the device that you’re running tests on? |
I only test on the Corellium builders. Running all.bash on devices is too painful for me. |
|
I've upgraded all builders now. I haven't reproduced the SIGILL crashes yet, but get these weird errors now: I've filed #46931. |
|
Thanks @eliasnaur and @markmentovai for the update! Regarding the workaround, I noticed that the SIGILL occurs very early, during bootstrapping (this is also the case you commented on CL 279489). It is possible that the workaround is actually effective, but the bootstrap toolchain, which does not include the workaround, may still crash. That said, as you said earlier the workaround only helps 14.0 - 14.4, so probably don't bother. Thanks. |
|
It's been a while we don't see this error on Corellium builders. It is either fixed on runtime already, or a problem on the previous builder itself (and we don't have them anymore). Shall we close this? |
|
Yeah, it's been a while we haven't seen this, since the new builders are set up. I think we can close. If this happens again we can reopen. Thanks. |
https://build.golang.org/log/8da8b3d360f7b226bbc021f29c3d9617d35773f9
The text was updated successfully, but these errors were encountered: