Release-blocker via #11811 (CC @golang/release).

@changkun and/or @steeve: could you investigate the source of these failures?

Just a first impression: I haven't touched any android builders yet but only iOS builders, and probably @steeve didn't too for years. If the failures occur later in the past three days, they may likely be caused by newly landed commits.

Happened a few more times and then stopped:

2021-12-13T22:45:26-5b9207f/android-arm64-corellium (This one got past bootstrap and then failed during test builds)
2021-12-13T18:41:35-083ef54/android-arm-corellium
2021-12-13T16:44:13-acc65b4/android-arm-corellium
2021-12-13T16:42:31-4b3d8d1/android-arm-corellium

It seems possible that the cmd/compile/internal/ssa package just got big enough that the build is now getting OOM-killed. The next step here may be to reproduce and look at the syslog.

Two more, also building cmd/compile/internal/ssa:

greplogs --dashboard -md -l -e '(?ms)\Aandroid-arm.*/compile: signal: killed' --since=2022-01-04

2022-01-18T23:59:40-50869f3/android-arm-corellium
2022-01-18T21:43:02-cf5d73e/android-arm-corellium

cmd/compile/internal/ssa is the largest and slowest package. This has to be the OOM killer.

2022-01-20T01:03:13-e7d5857/android-arm-corellium
2022-01-19T21:46:55-bb7fb8a/android-arm-corellium
2022-01-19T21:14:18-c1296af/android-arm64-corellium
2022-01-19T20:54:49-1efc581/android-arm-corellium
2022-01-19T16:33:05-3e45eb3/android-arm-corellium
2022-01-19T16:23:21-bec2cc3/android-arm-corellium
2022-01-19T15:34:05-ca33b34/android-arm-corellium

Change https://golang.org/cl/381514 mentions this issue: dashboard: set GOMAXPROCS=1 for android corellium builder

I hope we would revert the change. I've been contacting Corellium lately since before Christmas. There are multiple issues (including this) to their environment that needs to be fixed at least from what I have experienced:

1. The devices may be automatically turned off for an unknown reason (their response is under investigation)
2. The devices may be automatically being wiped out for an unknown reason (same response)
3. The VPN connection may not be established due to cert error

The impression is their environment monitor may trigger certain action (when OOM), then the entire device pool will be rescheduled or removed from running status, which causes frequent device missing from the farmer when there is no action being made from the human side.

@changkun Sorry, which change do you think we should revert?

@changkun Sorry, which change do you think we should revert?

Not sure what else I could refer to? It looks like there is only one change related to this thread. I was telling a story that the problem could be on the Corellium side based on the recent experience to maintain them. Hence we may not need to set GOMAXPROCS=1 as long as they confirmed and fixed it.

We'd be happy to be able to revert and run with more parallelism, but at least to me it's not clear how this is related to the Corellium issues you mentioned. From the build logs, this is not a device disappearing or a VPN issue. The compiler is getting a signal and that's successfully being reported up through all.bash and terminating the build. Maybe I'm not understanding what you're saying?

@changkun I'm sorry for being slow to understand. When I ask a specific question like "what change do you think we should revert" it really helps me a lot if you can simply provide a CL number or a git revision or something. Please feel free to also explain why it is a stupid question, but please also answer the question. Many thanks.

I'm going to guess that you mean https://go.dev/cl/381514? I don't know if that change is helping but I don't understand why reverting it would help either.

My apologies if I'm missing the point.