Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/env/darwin: one of our 10 Macs is down #35996

Closed
bradfitz opened this issue Dec 5, 2019 · 4 comments
Closed

x/build/env/darwin: one of our 10 Macs is down #35996

bradfitz opened this issue Dec 5, 2019 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-Darwin
Milestone
Unreleased

Comments

@bradfitz
Copy link
Contributor

bradfitz commented Dec 5, 2019

https://farmer.golang.org/#health says:

macstadium_host10a missing, not seen for 11h25m2s
macstadium_host10b missing, not seen for 11h22m30s
2 machines missing, 10% of capacity
makemac daemon: vm.destroy("mac_10_12_host10a") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.
makemac daemon: vm.destroy("mac_10_15_host10b") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.

Looks like we lost machine 10.

This is another case that would've been handled automatically by #32033 probably. (But this particular bug is about getting it healthy again)
@bradfitz bradfitz added OS-Darwin Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. labels Dec 5, 2019
@gopherbot gopherbot added this to the Unreleased milestone Dec 5, 2019
@cagedmantis
Copy link
Contributor

I've contacted MacStadium and requested remote hands for:
macstadium_host10a
macstadium_host10b

@bradfitz
Copy link
Contributor Author

bradfitz commented Dec 5, 2019

They won't know what those mean probably, as those are VM names, not host names.

Also, they have an API for this (#32033) and also a web UI, both letting us do this without their help:

https://portal.macstadium.com/subscriptions

To figure out which IP it is, you can look at https://10.88.203.9/ui/ (see https://github.com/golang/build/tree/master/env/darwin/macstadium)

@cagedmantis
Copy link
Contributor

The ticket I filed with MacStadium requests remote hands for three machines:
MacMini-5 [Network connectivity lost]
MacMini-6 [Network connectivity lost]
MacMini-10 [Not responding/ Cannot synchronize host]

While [MacMini-]5,6 seem to be back now, MacMini-10 is not.

@cagedmantis
Copy link
Contributor

All three machines are now operational.

@golang golang locked and limited conversation to collaborators Dec 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-Darwin
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@bradfitz @cagedmantis @gopherbot