Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/maintner: poll Gerrit more often than 5 minutes #21192

Open
bradfitz opened this issue Jul 27, 2017 · 7 comments
Open

x/build/maintner: poll Gerrit more often than 5 minutes #21192

bradfitz opened this issue Jul 27, 2017 · 7 comments
Labels
Builders x/build issues (builders, bots, dashboards)
Milestone
Unreleased

Comments

@bradfitz
Copy link
Contributor

If Gerrit emails are delayed (there's no SLA?), maintner (our Gerrit+GitHub mirror) doesn't get updated until the next Gerrit poll every 15 minutes.

We should probably do a cheap poll more often than 15 minutes.

I just saw an email delay of 3-5 minutes.

@andybons, what's a cheap (for Gerrit, and for our quota) way to ask Gerrit what's changed recently, including comments & votes?

A full git ls-remote on each project is pretty expensive, both in network & since we need to do it for each project.

A query over all changes/projects with a Limit of 1 seems cheap network-wise, but not sure what the quota is for that. Can we do it every minute?
@gopherbot gopherbot added this to the Unreleased milestone Jul 27, 2017
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jul 27, 2017
@andybons andybons self-assigned this Aug 21, 2017
@andybons
Copy link
Member

The query over all changes/projects with a Limit of 1 is the best way forward, but unauthenticated requests to Gerrit are capped at 0.2 qps over a day, while authenticated requests are at 1 qps over a day. So, if unauthenticated, we could have a max of 12 clients hitting Gerrit at 1 query per minute. I think that would be fine for now, but it’d get bad as more things are moved/built using maintner.

Might as well look into authenticated requests, then we can reduce the poll interval.

@dborowitz
Copy link
Contributor

As I told Andy privately, I strongly recommend setting up authentication. If you run over your quota, we can increase it, but only on a per-user basis. So you may find life easier if you're not scrambling to go from anonymous to authenticated while in the middle of an outage.

I'm slightly curious if you reduce the polling interval to 1 minute, how many intervals you will actually observe no updates to any changes. It might just be worth doing a "full" poll every 1 minute, not a "quick" poll to see if anything's changed, followed by a 100% chance of a full poll.

@andybons
Copy link
Member

We can do this rather easily with our instances using https://gerrit.googlesource.com/gcompute-tools/. Will start by tackling that.

@gopherbot
Copy link

Change https://golang.org/cl/57710 mentions this issue: maintner: decrease poll interval from 15 to 5 min

gopherbot pushed a commit to golang/build that referenced this issue Aug 21, 2017
@andybons
maintner: decrease poll interval from 15 to 5 min
f7c15b6 
When Gerrit emails are not sent, maintner polls Gerrit every
15 minutes. We can safely reduce this to every 5 min given the
relatively small number of bots using maintner and the current
quota of 0.2 qps per day that Gerrit permits for unauthenticated
requests.

Update golang/go#21192

Change-Id: I233bc50998927daedb34d81436e182579d2b4031
Reviewed-on: https://go-review.googlesource.com/57710
Reviewed-by: Sarah Adams <shadams@google.com>
@bradfitz
Copy link
Contributor Author

I would not use gcompute-tools. Many of our Docker containers only have a single process and don't even contain a filesystem. Just use the GCE metadata service like we do already. In fact, Gerrit auth credentials are already in there and used by some Gerrit client usages. We just need to use it in more places.

@andybons
Copy link
Member

I don’t see Gerrit auth credentials in the instance metadata. It is named something not obvious?

@bradfitz
Copy link
Contributor Author

Not on corp so I can't check. But gopherbot posts to Gerrit, so gopherbot is doing getting Gerrit auth somehow.

One of these three: https://github.com/golang/build/blob/158d74e/cmd/gopherbot/gopherbot.go#L76

@andybons andybons removed their assignment Sep 18, 2019
@andybons andybons changed the title x/build/maintner: poll Gerrit more often than 15 minutes x/build/maintner: poll Gerrit more often than 5 minutes Oct 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards)
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@bradfitz @dborowitz @andybons @gopherbot