-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: GitHub rate limit exceeded for various tools #31919
Comments
I've disabled gopherbot, gitmirror, and gerritbot for now. There is a repeated error from gopherbot I'm looking into. |
Issue #28320 might be related. |
Change https://golang.org/cl/176037 mentions this issue: |
maintnerd looks to be the reason why our quota keeps getting exhausted, as it got stuck in a loop. We're out of quota again so it's another hour or so at least before that and any other dependent services are available. |
We've identified this is connected to the incorrect timing information recorded on some GitHub comments posted a few hours ago. GitHub has said the issue has been fixed, and they're in process of correcting inaccurate timestamps: https://twitter.com/githubstatus/status/1126171459255185414 The current inaccurate timestamps are causing maintnerd to take incorrect actions and use up its rate limit quota. After another hour passes (unless GitHub corrects the incorrect timestamps even sooner), those comments will no longer be in the future and maintnerd should be able to run okay. So we're currently waiting a bit more. |
@dmitshur, did you figure out what the problem in maintnerd was? I saw this log line a bunch: func (p *githubRepoPoller) syncComments(ctx context.Context) error {
for {
nums := p.issueNumbersWithStaleCommentSync()
if len(nums) == 0 {
return nil
}
remain := len(nums)
for _, num := range nums {
p.logf("comment sync: %d issues remaining; syncing issue %v", remain, num) But I never saw the log from the code that immediately follows: if err := p.syncCommentsOnIssue(ctx, num); err != nil {
p.logf("comment sync on issue %d: %v", num, err)
return err
}
remain--
}
}
} So for n, gi := range p.gr.issues {
if !gi.commentsSynced() {
issueNums = append(issueNums, n)
}
} And func (gi *GitHubIssue) commentsSynced() bool {
if gi.NotExist {
// Issue doesn't exist, so can't sync its non-issues,
// so consider it done.
return true
}
return gi.commentsSyncedAsOf.After(gi.Updated)
} That commentsSyncedAsOf time.Time // as of server's Date header That's written to from: func (c *Corpus) processGithubIssueMutation(m *maintpb.GithubIssueMutation) {
....
if m.CommentStatus != nil && m.CommentStatus.ServerDate != nil {
if serverDate, err := ptypes.Timestamp(m.CommentStatus.ServerDate); err == nil {
gi.commentsSyncedAsOf = serverDate.UTC()
} But why didn't make progress before in ... func (p *githubRepoPoller) syncCommentsOnIssue(ctx context.Context, issueNum int32) error { Because of this GitHubIssue field? commentsUpdatedTil time.Time // max comment modtime seen And syncCommentsOnIssue does: since := issue.commentsUpdatedTil
p.c.mu.RUnlock()
owner, repo := p.gr.id.Owner, p.gr.id.Repo
morePages := true // at least try the first. might be empty.
for morePages {
ics, res, err := p.githubDirect.Issues.ListComments(ctx, owner, repo, int(issueNum), &github.IssueListCommentsOptions{
Since: since,
Direction: "asc",
Sort: "updated",
ListOptions: github.ListOptions{PerPage: 100},
}) .... but, uh, where is
Never? Um. |
I misread one of the lines as |
mergeIDs makes the assumption that the first email seen for a gopher is the one used for Gerrit. In the last update to gophers.go, the first email for Filippo was changed to be invalid, causing errors when automatic assignment was attempted in gopherbot. Update golang/go#31919 Change-Id: Ief9115d1aed8ab998e92cfed792444d1a6463df8 Reviewed-on: https://go-review.googlesource.com/c/build/+/176037 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
We're in the process of turning on |
We've recovered all services from this outage by now. |
Change https://golang.org/cl/195062 mentions this issue: |
This is a cherry-pick of CL 135456 that restores Gerrit emails for some people that were incorrectly changed in CL 165639, with manual no-op addPerson line merges to address code review comments. The cherry-pick applied very cleanly with just two minor merge conflicts: one due to a Filippo's email already being fixed in CL 176037, and another due to a close new entry. Add tests to catch this from happening again, until the source of the problem is resolved in issue golang/go#34259. Updates golang/go#34259 Updates golang/go#28320 Updates golang/go#31919 Updates golang/go#27517 Updates golang/go#27631 Change-Id: Ia03a2b94403334d3f571ac5623e12d3bfd6f1e4f Reviewed-on: https://go-review.googlesource.com/c/build/+/195062 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This is a cherry-pick of CL 135456 that restores Gerrit emails for some people that were incorrectly changed in CL 165639, with manual no-op addPerson line merges to address code review comments. The cherry-pick applied very cleanly with just two minor merge conflicts: one due to a Filippo's email already being fixed in CL 176037, and another due to a close new entry. Add tests to catch this from happening again, until the source of the problem is resolved in issue golang/go#34259. Updates golang/go#34259 Updates golang/go#28320 Updates golang/go#31919 Updates golang/go#27517 Updates golang/go#27631 Change-Id: Ia03a2b94403334d3f571ac5623e12d3bfd6f1e4f Reviewed-on: https://go-review.googlesource.com/c/build/+/195062 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This specific outage has been resolved, and this issue isn't actionable, so I'll close it. We can open new issues if there is specific work that needs to be done. |
gopherbot can't make any calls to GitHub and gerritbot logs are showing that our quota is getting drained very quickly.
@bradfitz @dmitshur
The text was updated successfully, but these errors were encountered: