Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: 2 darwin builders are missing #41280

Closed
dmitshur opened this issue Sep 9, 2020 · 1 comment
Closed

x/build: 2 darwin builders are missing #41280

dmitshur opened this issue Sep 9, 2020 · 1 comment
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Sep 9, 2020

From https://farmer.golang.org/#health:

# "macs" status: MacStadium Mac VMs
# Notes: https://github.com/golang/build/tree/master/env/darwin/macstadium
Warn: macstadium_host08a missing, not seen for 55h42m8s
Warn: macstadium_host08b missing, not seen for 55h42m6s
Warn: 2 machines missing, 10% of capacity
Warn: makemac daemon: vm.destroy("mac_10_12_host08a") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.

Warn: makemac daemon: vm.destroy("mac_10_12_host08b") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.

The machine hosting macstadium_host08a and macstadium_host08b is being unresponsive.

I've taken some initial steps to bring it back up, but this might need more work. I'll resume tomorrow.

/cc @toothrot @andybons

@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Sep 9, 2020
@dmitshur dmitshur added this to the Unreleased milestone Sep 9, 2020
@dmitshur dmitshur self-assigned this Sep 9, 2020
@dmitshur
Copy link
Contributor Author

dmitshur commented Sep 9, 2020

This is fixed now.

I've taken some initial steps to bring it back up, but this might need more work.

I could see that the host was being unresponsive in the VMware web UI. Attempts to restart it there were unsuccessful. However, it helped to restart the physical node via the https://portal.macstadium.com/subscriptions UI (4th bullet point here), then it became possible to get it back into healthy state in the VMware cluster.

@dmitshur dmitshur closed this as completed Sep 9, 2020
@golang golang locked and limited conversation to collaborators Sep 9, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

2 participants