Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: Mac VMs down #23441

Closed
bradfitz opened this issue Jan 14, 2018 · 2 comments
Closed

x/build: Mac VMs down #23441

bradfitz opened this issue Jan 14, 2018 · 2 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@bradfitz
Copy link
Contributor

bradfitz commented Jan 14, 2018 •
edited

I'm on vacation for 2 weeks, but I peeked at why the Mac builders are dead and trybots are thus hanging.

As background, the Macs builders are 10 physical Mac Minis hosted at MacStadium.com, running a VMware vSphere cluster, where we run up to 20 virtual macs on those 10 physical macs.

The Linux box & Go daemon that runs the show is a Linux VM on the same vSphere cluster. Its host name is macstadiumd.golang.org. It's not responding to pings.

My guess is MacStadium updated our vSphere cluster for us for the Spectre/Meltdown patches, which requires a full reboot of all VMware hosts & VMs. (I just had to do this myself for my home vSphere setup)

And I further suspect that our macstadiumd.golang.org VM was never marked to automatically boot on vSphere power-up.

(A previous bug, #20224, was about making sure the daemons on that machine were in systemd at least so they started automatically upon boot, but I guess we forgot to make sure that the machine itself powers on.)

@rsc has the credentials & VPN instructions for getting into the vSphere admin web UI. (be sure to enable Flash in Chrome Content Settings!)

/cc @andybons @rsc @ianlancetaylor
@bradfitz bradfitz added Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. labels Jan 14, 2018
@gopherbot gopherbot added this to the Unreleased milestone Jan 14, 2018
@rsc
Copy link
Contributor

rsc commented Jan 15, 2018

The Linux system had a blank console unresponsive to input. I powered off and powered on the Linux system. Now it has a login prompt, and the Mac VMs started disappearing and reappearing, I assume under the Linux system's management.

@rsc
Copy link
Contributor

rsc commented Jan 15, 2018

The Jan 14 commit on the main dashboard has filled in all the darwin holes it had. A new darwin column for 10.8 has appeared as well - it has just one ok in it now, but I assume that's going to get filled in more as time goes on, or at least is different from this machine being hung. Going to assume the trybots are unstuck now too and close this.

@rsc rsc closed this as completed Jan 15, 2018
@golang golang locked and limited conversation to collaborators Jan 15, 2019
@rsc rsc removed their assignment Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@bradfitz @rsc @gopherbot