-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: corellium builders failing with signal: killed #54819
Comments
There was no manual intervention on the builder side to my knowledge. It is either caused by internal service changes on the Corellium side or the changes in the Go code base. There is not much we can do for the first case, although I've communicated with them many times in the past. For the second case, if the build process somehow changes to utilize more resources, such as memory, it may hit the VM limit set by Corellium, which could lead to the build failure. |
cc @cuonglm as CL 421880 seems to be the first commit that causes: https://build.golang.org/log/b787fac76217446a7796bfe687e299e2313baf67 |
They look like OOM kill. How much memory does the builder have? Does the build run in parallel with other programs or containers? |
These are the system info collected using Android Builder 1:
Android Builder 2:
Android Builder 3:
|
This is a real failed and was fixed at tip. The OOM killed seems different problem. |
Based on some recent observations, it seems that the issue is gone. Perhaps now Corellium internal becomes more stable in managing their VMs and not killing OOMed builders anymore. At some point, we may simply close this issue without any further action. |
Found new dashboard test flakes for:
2022-11-28 15:02 android-arm64-corellium go@cc7530da (log)
2022-11-28 16:06 android-arm-corellium go@76ec47ee (log)
2022-11-28 19:38 android-arm64-corellium go@bb0d8297 (log)
2022-12-15 17:45 android-arm64-corellium go@628a1e7d (log)
|
Found new dashboard test flakes for:
2022-11-28 17:25 android-arm-corellium go@0f28c478 (log)
2022-11-28 17:25 android-arm64-corellium go@0f28c478 (log)
2022-11-28 19:33 android-arm-corellium go@61f5a672 (log)
|
Found new dashboard test flakes for:
2023-01-03 18:52 android-arm-corellium go@cdc73f06 (log)
2023-01-03 18:52 android-arm64-corellium go@cdc73f06 (log)
2023-01-03 18:53 android-arm-corellium go@d03231d9 (log)
2023-01-03 18:53 android-arm64-corellium go@d03231d9 (log)
2023-01-03 20:31 android-arm-corellium go@9955a7e9 (log)
|
Found new dashboard test flakes for:
2023-01-16 08:37 android-arm-corellium go@1c65b69b (log)
2023-01-17 14:18 android-arm-corellium go@145dd384 (log)
2023-01-17 14:18 android-arm64-corellium go@145dd384 (log)
2023-01-17 15:26 android-arm-corellium go@66689c7d (log)
2023-01-17 16:11 android-arm64-corellium go@8409251e (log)
2023-01-17 16:25 android-arm64-corellium go@02ed0e5e (log)
2023-01-17 18:21 android-arm64-corellium go@9088c691 (log)
2023-01-17 22:13 android-arm64-corellium go@f773bef9 (log)
2023-01-17 22:30 android-arm-corellium go@dbe327a6 (log)
|
Found new dashboard test flakes for:
2023-01-30 16:00 android-arm-corellium go@d42c08a2 cmd/addr2line [build] (log)
2023-01-30 16:00 android-arm-corellium go@d42c08a2 cmd/asm [build] (log)
2023-01-30 16:00 android-arm-corellium go@d42c08a2 cmd/buildid [build] (log)
2023-01-30 16:00 android-arm-corellium go@d42c08a2 cmd/cgo [build] (log)
2023-01-30 16:00 android-arm-corellium go@d42c08a2 cmd/compile [build] (log)
2023-01-30 16:41 android-arm-corellium go@a34cec8e cmd/addr2line [build] (log)
2023-01-30 16:41 android-arm-corellium go@a34cec8e cmd/asm [build] (log)
2023-01-30 16:41 android-arm-corellium go@a34cec8e cmd/buildid [build] (log)
2023-01-30 16:41 android-arm-corellium go@a34cec8e cmd/cgo [build] (log)
2023-01-30 16:41 android-arm-corellium go@a34cec8e cmd/compile [build] (log)
2023-01-30 18:48 android-arm-corellium go@cb61585e cmd/addr2line [build] (log)
2023-01-30 18:48 android-arm-corellium go@cb61585e cmd/asm [build] (log)
2023-01-30 18:48 android-arm-corellium go@cb61585e cmd/buildid [build] (log)
2023-01-30 18:48 android-arm-corellium go@cb61585e cmd/cgo [build] (log)
2023-01-30 18:48 android-arm-corellium go@cb61585e cmd/compile [build] (log)
2023-01-30 18:49 android-arm64-corellium go@3875258f (log)
2023-01-31 15:09 android-arm64-corellium go@066b7805 (log)
2023-01-31 16:52 android-arm-corellium go@00e1091b (log)
|
Found new dashboard test flakes for:
2023-02-14 16:24 android-arm64-corellium go@b146d762 (log)
2023-02-14 16:51 android-arm64-corellium go@0af2c78c (log)
2023-02-15 15:30 android-arm64-corellium go@30151f2a (log)
2023-02-15 22:00 android-arm64-corellium go@ed370d87 (log)
2023-02-15 22:02 android-arm64-corellium go@2994e9aa (log)
2023-02-16 00:16 android-arm64-corellium go@55044288 (log)
2023-02-16 02:02 android-arm64-corellium go@518889b3 (log)
2023-02-16 19:34 android-arm64-corellium go@1a9893a9 (log)
2023-02-16 23:09 android-arm64-corellium go@3ad6393f (log)
2023-02-17 03:56 android-arm64-corellium go@21f43405 (log)
2023-02-17 20:31 android-arm64-corellium go@990501e7 (log)
2023-02-17 20:39 android-arm64-corellium go@37e893b1 (log)
2023-02-17 21:23 android-arm64-corellium go@209df389 (log)
2023-02-17 22:19 android-arm-corellium go@d3daeb52 (log)
2023-02-17 22:21 android-arm-corellium go@d49719b1 (log)
2023-02-17 22:21 android-arm64-corellium go@d49719b1 (log)
2023-02-17 23:54 android-arm-corellium go@7e5906a5 (log)
2023-02-17 23:54 android-arm64-corellium go@7e5906a5 (log)
|
Found new dashboard test flakes for:
2023-03-10 22:47 android-arm-corellium go@d9c29ec6 (log)
|
Found new dashboard test flakes for:
2023-03-13 14:16 android-arm64-corellium go@b561ebab (log)
|
Found new dashboard test flakes for:
2023-03-13 14:16 android-arm-corellium go@b561ebab (log)
2023-03-13 14:20 android-arm-corellium go@b6f29d23 (log)
|
Found new dashboard test flakes for:
2023-03-13 15:28 android-arm64-corellium go@43f911b0 (log)
2023-03-13 16:47 android-arm64-corellium go@91c82ff7 (log)
|
Found new dashboard test flakes for:
2023-03-13 17:06 android-arm64-corellium go@90dde5de (log)
|
Found new dashboard test flakes for:
2023-03-13 19:26 android-arm64-corellium go@7c019c62 (log)
|
Found new dashboard test flakes for:
2023-03-13 19:26 android-arm-corellium go@7c019c62 (log)
|
Found new dashboard test flakes for:
2023-03-28 16:22 android-arm64-corellium go@f7156f63 (log)
|
Found new dashboard test flakes for:
2023-05-22 19:05 android-arm64-corellium go@da2476a8 (log)
|
Found new dashboard test flakes for:
2023-05-22 19:37 android-arm64-corellium go@8c445b7c (log)
|
Found new dashboard test flakes for:
2023-05-22 20:56 android-arm-corellium go@5abfdc8c (log)
|
Found new dashboard test flakes for:
2023-05-22 19:03 android-arm64-corellium go@b66474e0 (log)
|
Found new dashboard test flakes for:
2023-05-22 18:27 android-arm-corellium go@bc96901e (log)
|
Found new dashboard test flakes for:
2023-05-22 17:31 android-arm-corellium go@1dad7ef5 (log)
|
Found new dashboard test flakes for:
2023-05-22 22:47 android-arm-corellium go@7f1467ff (log)
|
Found new dashboard test flakes for:
2023-05-22 16:55 android-arm64-corellium go@d2db8267 (log)
|
Found new dashboard test flakes for:
2023-05-23 02:43 android-arm64-corellium go@88704b18 (log)
|
Found new dashboard test flakes for:
2023-05-23 11:36 android-arm64-corellium go@0cd04724 (log)
|
Found new dashboard test flakes for:
2023-05-23 11:37 android-arm-corellium go@f81c8858 (log)
|
Found new dashboard test flakes for:
2023-05-23 11:36 android-arm-corellium go@380529d5 (log)
|
Found new dashboard test flakes for:
2023-05-23 13:22 android-arm64-corellium go@40bdc561 (log)
|
Found new dashboard test flakes for:
2023-05-23 16:36 android-arm-corellium go@876eca79 (log)
2023-05-23 18:15 android-arm-corellium go@b0f15b4a (log)
|
Found new dashboard test flakes for:
2023-05-23 19:39 android-arm-corellium go@0aedfc19 (log)
|
Found new dashboard test flakes for:
2023-05-23 17:31 android-arm64-corellium go@45b74d20 (log)
|
Found new dashboard test flakes for:
2023-05-23 22:13 android-arm64-corellium go@6b7aab77 (log)
2023-05-23 23:33 android-arm64-corellium go@0df68124 (log)
2023-05-24 00:38 android-arm64-corellium go@392fbaa0 (log)
|
Found new dashboard test flakes for:
2023-05-23 20:55 android-arm-corellium go@c4afec23 (log)
2023-05-23 21:35 android-arm-corellium go@94c75232 (log)
2023-05-23 22:13 android-arm-corellium go@6b7aab77 (log)
2023-05-23 23:33 android-arm-corellium go@0df68124 (log)
|
Found new dashboard test flakes for:
2023-06-15 21:14 android-arm64-corellium go@30b17f4f (log)
|
Found new dashboard test flakes for:
2023-06-16 14:09 android-arm64-corellium go@1a7709d6 (log)
|
Found new dashboard test flakes for:
2023-06-16 17:08 android-arm-corellium go@2b0ff4b6 (log)
|
Found new dashboard test flakes for:
2023-06-19 20:30 android-arm64-corellium go@bc21d6a4 (log)
2023-06-20 00:47 android-arm64-corellium go@98617fd2 (log)
|
Found new dashboard test flakes for:
2023-06-20 00:47 android-arm-corellium go@98617fd2 (log)
|
Found new dashboard test flakes for:
2023-07-05 15:21 android-arm64-corellium go@b2215e49 (log)
|
Found new dashboard test flakes for:
2023-07-05 22:05 android-arm64-corellium go@c8dad424 (log)
2023-07-05 22:05 android-arm-corellium go@3fce1115 (log)
2023-07-05 22:05 android-arm64-corellium go@3fce1115 (log)
|
Found new dashboard test flakes for:
2023-07-06 13:09 android-arm64-corellium go@b490bdc2 (log)
2023-07-06 13:09 android-arm-corellium go@6305d7fd (log)
2023-07-06 13:09 android-arm64-corellium go@6305d7fd (log)
2023-07-06 19:37 android-arm-corellium go@449ef379 (log)
2023-07-06 19:37 android-arm64-corellium go@449ef379 (log)
|
Found new dashboard test flakes for:
2023-07-07 15:07 android-arm-corellium go@d3d78b4b (log)
|
Found new dashboard test flakes for:
2023-07-06 23:23 android-arm64-corellium go@39c50707 (log)
2023-07-07 15:07 android-arm64-corellium go@d3d78b4b (log)
2023-07-07 17:39 android-arm64-corellium go@158d1119 (log)
|
Found new dashboard test flakes for:
2023-07-07 20:48 android-arm64-corellium go@894d24d6 (log)
2023-07-07 22:02 android-arm64-corellium go@5c154986 (log)
2023-07-10 16:36 android-arm64-corellium go@7dc62f3b (log)
|
Found new dashboard test flakes for:
2023-07-07 20:48 android-arm-corellium go@894d24d6 (log)
2023-07-07 22:02 android-arm-corellium go@5c154986 (log)
2023-07-10 16:29 android-arm-corellium go@c4db811e (log)
2023-07-10 16:36 android-arm-corellium go@7dc62f3b (log)
|
2022-09-01T15:27:07-b5b2cf5/android-arm-corellium
2022-09-01T02:24:14-91ef076/android-arm-corellium
2022-09-01T00:42:27-64b260d/android-arm-corellium
2022-09-01T00:42:27-64b260d/android-arm64-corellium
2022-08-31T23:52:00-ca634fa/android-arm-corellium
2022-08-31T23:52:00-ca634fa/android-arm64-corellium
2022-08-31T22:22:43-e4b624e/android-arm-corellium
2022-08-31T22:09:44-5f5c018/android-arm-corellium
2022-08-31T22:04:55-af7f067/android-arm-corellium
2022-08-31T21:10:41-52d9e7f/android-arm64-corellium
2022-08-31T20:05:46-239115c/android-arm-corellium
2022-08-31T20:05:06-9085ff5/android-arm64-corellium
2022-08-31T16:37:20-ebc75ac/android-arm64-corellium
2022-08-31T15:33:00-fa0e3bf/android-arm-corellium
2022-08-31T15:30:57-d01200e/android-arm-corellium
2022-08-31T15:30:26-a25a34a/android-arm64-corellium
2022-08-31T15:12:21-d3b35a4/android-arm64-corellium
2022-08-31T15:12:18-28e3885/android-arm64-corellium
2022-08-31T09:35:10-d2e0587/android-arm-corellium
2022-08-31T01:16:54-ee0e40a/android-arm-corellium
cc @golang/android
The text was updated successfully, but these errors were encountered: