Our MacStadium VMware host machines have a tendency to die or somehow wedge and stop reporting to vSphere.

MacStadium has an API now to hard power cycle machines:

https://www.macstadium.com/features/api

Sever Action (Power) by IP
URI: https://api.macstadium.com/core/api/servers/{id or ip_address}/{action}
Request: GET
Params: Action can be ‘On’,’Off’, or ‘Reboot’ (case insensitive).
Return: String. ‘True’ on success or an error message.
Example:

curl -H "Accept: application/json" -H "Content-Type: application/json" -X GET -u user:pass https://api.macstadium.com/core/api/servers/192.168.1.101/reboot

We should make our x/build/cmd/makemac daemon monitor the healthy status of our 10 VMware nodes (using the govc API or just whether they've registered with the coordinator in the past N minutes) and if we suspect a node is dead, we hard-reboot it with the API.

/cc @andybons @dmitshur @bcmills