-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: update k8s clusters #33529
Comments
[Development] successfully disabled Kubernetes Dashboard
|
[Production] successfully disabled Kubernetes Dashboard add-on
|
Stepping away until 4pm ET. Will begin upgrade of k8s clusters then. |
[Dev] No issues found so far. Will continue with prod tomorrow morning. |
Delayed, but upgrading prod cluster now. |
Updated masters and nodes to 1.12.7-gke.25 Scaleway builders are having issues. From https://farmer.golang.org:
Investigating |
From the scaleway machine (
|
Cleaned up stopped container noted in that output. Will continue on other machines. |
All Scaleway Docker images are back up now and talking to coordinator. No other issues seen. Will reopen bug if another issue comes up. |
It tries to remove exited containers here: https://github.com/golang/build/blob/master/cmd/rundockerbuildlet/rundockerbuildlet.go#L91-L93 It also tries to remove "Created" containers that never reach the running status here: https://github.com/golang/build/blob/master/cmd/rundockerbuildlet/rundockerbuildlet.go#L111-L114 Based on the log lines you posted, I would expect to see an error if it tried and failed to remove one of these containers. I'm guessing it's not detecting it for some reason. It's possible the container ended up in a status we're not handling: It's also possible that our logic in L91-L93 isn't fetching the status properly from the formatted string (an extra space in a name? multiple names?). It's hard to tell the root cause at this point as all the impacted hosts have been fixed manually, so we can't inspect the conflicting container status. |
Placeholder issue for upgrading our k8s clusters
The text was updated successfully, but these errors were encountered: