Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/makemake: Macs are hung trying to delete a VM #31212

Closed
bradfitz opened this issue Apr 2, 2019 · 4 comments
Closed

x/build/cmd/makemake: Macs are hung trying to delete a VM #31212

bradfitz opened this issue Apr 2, 2019 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Milestone

Comments

@bradfitz
Copy link
Contributor

bradfitz commented Apr 2, 2019

I noticed most the Macs are gone so I looked at the makemac daemon:

  533 ?        Ssl    0:00 /home/gopher/makemac -auto
  559 ?        Sl     0:00  \_ govc vm.destroy mac_10_12_host04a

It's been hung like that for some time.

Probably need a context there. But the cluster is probably unhealthy.

/cc @dmitshur

@gopherbot gopherbot added this to the Unreleased milestone Apr 2, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Apr 2, 2019
@bradfitz
Copy link
Contributor Author

bradfitz commented Apr 2, 2019

Yup, one of our 10 hosts is down.

Yay 10 single points of failure.

@bradfitz
Copy link
Contributor Author

bradfitz commented Apr 2, 2019

Oh, it does eventually learn that the host is down, and now the API call fails quickly:

gopher@godns:~$ time govc vm.destroy mac_10_12_host04a
govc: Unable to communicate with the remote host, since it is disconnected.

real	0m0.161s
user	0m0.040s
sys	0m0.004s
gopher@godns:~$ echo $?
1

So I guess we can just ignore failures in that case.

@bradfitz
Copy link
Contributor Author

bradfitz commented Apr 2, 2019

Well, we already ignore the failure in that case:

                        log.Printf("Destroying VM %q unknown to coordinator...", name)
                        err := govc(ctx, "vm.destroy", name)
                        log.Printf("vm.destroy(%q) = %v", name, err)
                        dirty = true
                }

(Where the dirty flag just causes an API call to list VMs again later.)

So we're good on makemac. And I have a support request open with MacStadium to fix that host.

@bradfitz
Copy link
Contributor Author

bradfitz commented Apr 2, 2019

And things are good again. For now.

@bradfitz bradfitz closed this as completed Apr 2, 2019
@golang golang locked and limited conversation to collaborators Apr 1, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Projects
None yet
Development

No branches or pull requests

2 participants