Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: android-arm[64]-corellium builders failing #57634

Closed
heschi opened this issue Jan 5, 2023 · 4 comments
Closed

x/build: android-arm[64]-corellium builders failing #57634

heschi opened this issue Jan 5, 2023 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-Android
Milestone
Unreleased

Comments

@heschi
Copy link
Contributor

heschi commented Jan 5, 2023

Builders have started failing with what I suspect are OOMs:

cmd/compile/internal/ssa: /data/data/com.termux/files/home/tmpdir/workdir-host-android-arm64-corellium-android/go/pkg/tool/android_arm64/compile: signal: killed

https://build.golang.org/log/83f319d46b9e488bfd89fa14ca9eaae28f910842
https://build.golang.org/log/e7eff7d04d7e6bc95c5802566a15819d53f02166

etc

cc @golang/android
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jan 5, 2023
@gopherbot gopherbot added this to the Unreleased milestone Jan 5, 2023
@heschi heschi added NeedsFix The path to resolution is known, but the work has not been done. and removed Builders x/build issues (builders, bots, dashboards) labels Jan 5, 2023
@bcmills bcmills added Builders x/build issues (builders, bots, dashboards) OS-Android labels Jan 5, 2023
@changkun
Copy link
Member

changkun commented Jan 5, 2023

I just rebooted the Android builders.

@heschi
Copy link
Contributor Author

heschi commented Jan 5, 2023

Looks better, thanks. Anything that should be done to keep it from reoccurring? Even just automatically rebooting every week or something?

@changkun
Copy link
Member

changkun commented Jan 5, 2023

According to past observation, it looks like, between Go 1.19 and 1.20, the compiler SSA uses much more memory and often hits the memory limit of the builders (2GB).

A scheduled reboot might be helpful, but it hides the problem.

To my knowledge, there is no operational API to reboot builders on the Corellium side. It would be helpful if one can investigate more and reduce the memory usage at bootstrap, maybe using runtime/debug.SetMemoryLimit.

@heschi
Copy link
Contributor Author

heschi commented Jan 5, 2023

Feel free to file an issue for reducing bootstrap memory usage. But it seems like there's also some kind of memory leak happening on the builder, and working around that would be nice too.

Doesn't sound like there's anything much else to do here. Closing.

@heschi heschi closed this as completed Jan 5, 2023
@changkun changkun mentioned this issue Jan 12, 2023
Closed
@golang golang locked and limited conversation to collaborators Jan 5, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-Android
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@bcmills @changkun @gopherbot @heschi