Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/gerritbot: running into letsencrypt authorization rate limit #37377

Closed
dmitshur opened this issue Feb 22, 2020 · 4 comments
Closed

x/build/cmd/gerritbot: running into letsencrypt authorization rate limit #37377

dmitshur opened this issue Feb 22, 2020 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

From gerritbot logs:

2020/02/22 15:50:23 Updating data from log *maintner.netMutSource ...
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: 429 urn:acme:error:rateLimited: Error creating new authz :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:55:39 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate
2020/02/22 15:56:21 http: TLS handshake error from x.y.z.u:w: acme/autocert: missing certificate

That seems like something that shouldn't happen.

I don't know what the effect of this is. It may be harmless, or it may not be.
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Feb 22, 2020
@dmitshur dmitshur added this to the Unreleased milestone Feb 22, 2020
@dmitshur
Copy link
Contributor Author

dmitshur commented Feb 22, 2020 •
edited

For now, I've just redeployed gerritbot. Let's see if this continues to happen.

Edit: It's still happening.

It may not be very important to fix, the only HTTP page gerritbot is serving seems to be this one:

https://github.com/golang/build/blob/625f09dad308b2c1ef6791e1a4237c62027aaedf/cmd/gerritbot/gerritbot.go#L175-L178

But we should still fix it so it doesn't bother Let's Encrypt unnecessarily and doesn't spam our logs.

@cagedmantis
Copy link
Contributor

This failure should be visible in a health check as well. I've just encountered another instance of this happening.

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Sep 8, 2021
@dmitshur
Copy link
Contributor Author

dmitshur commented Sep 8, 2021

The problem turned out to be a misconfigured use of autocert (not enabling acme.ALPNProto, and not using HTTPHandler), which prevented gerritbot's HTML "hello" page from being served. CL 348433 fixes it. CC @heschi.

@heschi heschi closed this as completed Sep 8, 2021
@heschi heschi reopened this Sep 8, 2021
@gopherbot
Copy link

Change https://golang.org/cl/348433 mentions this issue: cmd/gerritbot: move to Workload Identity

@rsc rsc unassigned heschi Jun 23, 2022
@golang golang locked and limited conversation to collaborators Jun 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@cagedmantis @dmitshur @gopherbot @heschi