Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: LUCI presubmit outage due to build starvation #64429

Closed
mknyszek opened this issue Nov 28, 2023 · 2 comments
Closed

x/build: LUCI presubmit outage due to build starvation #64429

mknyszek opened this issue Nov 28, 2023 · 2 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Soon This needs to be done soon. (regressions, serious bugs, outages)
Milestone
Unreleased

Comments

@mknyszek
Copy link
Contributor

LUCI presubmit builds are pending a long time (hours) because our VMs aren't showing up in Swarming. We've got one VM chugging a long but presubmit is unusable at the moment.

We're investigating and have escalated the issue as well. Please use Run-TryBots+1 instead of Commit-Queue+1 in the interim. We'll update this issue once we've figured out the root cause and again once the issue is resolved.

Thanks.
@mknyszek mknyszek added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 28, 2023
@mknyszek mknyszek added this to the Unreleased milestone Nov 28, 2023
@heschi heschi pinned this issue Nov 28, 2023
@dmitshur dmitshur added the Soon This needs to be done soon. (regressions, serious bugs, outages) label Nov 28, 2023
@mknyszek
Copy link
Contributor Author

The root cause was discovered on the LUCI side and rolled back. Our bots are back online and the queue is getting processed. It may take a little while to recover, but our pool is scaling up to handle the demand so it shouldn't be too long.

@mknyszek
Copy link
Contributor Author

The queue of pending builds has drained. Calling this fixed.

@mknyszek mknyszek unpinned this issue Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Soon This needs to be done soon. (regressions, serious bugs, outages)
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
2 participants
@mknyszek @dmitshur