-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: gitmirror or maintnerd consumed all our Gerrit quota #23853
Comments
Our ls-remote quota has been reset. After investigating with @bradfitz, the requests were coming from either mainternd or gitmirror, not GerritBot. |
Sorry, I by default blame whatever changed last. I'm still debugging. |
Looking at logs in the Cloud Logging console (which captures all the GKE pod output), started at "2018-02-15T13:27:16.527663505Z" , or 5:27am Pacific: "2018-02-15 05:27:14.000 PST" I'll look at logs just before that to see if it's obvious what went crazy. |
Based on the error (presenting the IP address), we’re not authenticating our git requests somewhere. If we do this then we’ll get better error logging and much higher quota. |
@andybons, will do. We historically never did, because we were bound to 1 IP address and got high quota for that IP. Now that we bounce around k8s nodes, I'll make them authenticate. |
Change https://golang.org/cl/94836 mentions this issue: |
From another bug: We recently upgraded our Kubernetes cluster and our 10 random container jobs re-laid themselves out onto our 4 physical nodes (i.e. 4 egress IP addresses) We got lucky before and our 3 gerrit-hitting jobs were using different nodes (different IPs). But after this latest upgrade, all 3 are on the same node, so we burn through that IP's quota by the end of the day. That's the current theory. I'm adding auth now. |
Then use it from gitmirror and maintnerd. Updates golang/go#23853 Change-Id: I8112f004638667894676c04fa218a7ced10422ac Reviewed-on: https://go-review.googlesource.com/94836 Reviewed-by: Andrew Bonventre <andybons@golang.org>
This happened. |
Maintner is blocked:
So now all bots are down.
GerritBot consumed all of our Gerrit quota.
@andybons, please stop GerritBot and/or increase our Gerrit quota.
The text was updated successfully, but these errors were encountered: