-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build,cmd/compile: frequent "signal killed" building cmd/compile/internal/ssa on android-arm-corellium builder since 2021-12-07 #50084
Comments
Just a first impression: I haven't touched any android builders yet but only iOS builders, and probably @steeve didn't too for years. If the failures occur later in the past three days, they may likely be caused by newly landed commits. |
Happened a few more times and then stopped: 2021-12-13T22:45:26-5b9207f/android-arm64-corellium (This one got past bootstrap and then failed during test builds) |
It seems possible that the cmd/compile/internal/ssa package just got big enough that the build is now getting OOM-killed. The next step here may be to reproduce and look at the syslog. |
Two more, also building
2022-01-18T23:59:40-50869f3/android-arm-corellium |
cmd/compile/internal/ssa is the largest and slowest package. This has to be the OOM killer. 2022-01-20T01:03:13-e7d5857/android-arm-corellium |
Change https://golang.org/cl/381514 mentions this issue: |
I hope we would revert the change. I've been contacting Corellium lately since before Christmas. There are multiple issues (including this) to their environment that needs to be fixed at least from what I have experienced:
The impression is their environment monitor may trigger certain action (when OOM), then the entire device pool will be rescheduled or removed from running status, which causes frequent device missing from the farmer when there is no action being made from the human side. |
@changkun Sorry, which change do you think we should revert? |
Not sure what else I could refer to? It looks like there is only one change related to this thread. I was telling a story that the problem could be on the Corellium side based on the recent experience to maintain them. Hence we may not need to set GOMAXPROCS=1 as long as they confirmed and fixed it. |
We'd be happy to be able to revert and run with more parallelism, but at least to me it's not clear how this is related to the Corellium issues you mentioned. From the build logs, this is not a device disappearing or a VPN issue. The compiler is getting a signal and that's successfully being reported up through all.bash and terminating the build. Maybe I'm not understanding what you're saying? |
@changkun I'm sorry for being slow to understand. When I ask a specific question like "what change do you think we should revert" it really helps me a lot if you can simply provide a CL number or a git revision or something. Please feel free to also explain why it is a stupid question, but please also answer the question. Many thanks. I'm going to guess that you mean https://go.dev/cl/381514? I don't know if that change is helping but I don't understand why reverting it would help either. My apologies if I'm missing the point. |
greplogs --dashboard -md -l -e '(?ms)\Aandroid-arm.*/compile: signal: killed'
2021-12-08T18:06:06-7b7efd7/android-arm-corellium
2021-12-08T17:04:27-c759ec2/android-arm-corellium
2021-12-07T21:54:30-cf1ec17/android-arm-corellium
2021-12-07T17:46:04-4c943ab/android-arm-corellium
(Note 2-year gap!)
2019-11-14T00:58:33-e77106c/android-arm64-corellium
2019-11-13T20:53:39-49e05d4/android-arm64-corellium
2019-11-13T19:15:27-7ad2748/android-arm64-corellium
2019-06-18T20:32:15-2ba9f7c/android-arm-corellium
The text was updated successfully, but these errors were encountered: