Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: some Gerrit CLs are getting into a state where they can't be updated #40147

Open
dmitshur opened this issue Jul 10, 2020 · 4 comments
Open

x/build: some Gerrit CLs are getting into a state where they can't be updated #40147

dmitshur opened this issue Jul 10, 2020 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

From gopherbot logs, there are a number of CLs where a reviewer fails to be added due to the error "Change nnn may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.":

2020/07/10 13:27:29 No reviewers or cc: https://go-review.googlesource.com/c/text/+/238119
2020/07/10 13:27:29 Setting review on https://go-review.googlesource.com/c/text/+/238119: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:mpvl@golang.org State:}]}
2020/07/10 13:27:29 Could not set review for change "text~238119": HTTP status 409 Conflict; Change 238119 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:29 No reviewers or cc: https://go-review.googlesource.com/c/mobile/+/212839
2020/07/10 13:27:29 Setting review on https://go-review.googlesource.com/c/mobile/+/212839: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:hyangah@gmail.com State:}]}
2020/07/10 13:27:29 Could not set review for change "mobile~212839": HTTP status 409 Conflict; Change 212839 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:29 No reviewers or cc: https://go-review.googlesource.com/c/mobile/+/231681
2020/07/10 13:27:29 Setting review on https://go-review.googlesource.com/c/mobile/+/231681: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:hyangah@gmail.com State:}]}
2020/07/10 13:27:29 Could not set review for change "mobile~231681": HTTP status 409 Conflict; Change 231681 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:29 No reviewers or cc: https://go-review.googlesource.com/c/mobile/+/234239
2020/07/10 13:27:29 Setting review on https://go-review.googlesource.com/c/mobile/+/234239: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:hyangah@gmail.com State:}]}
2020/07/10 13:27:29 Could not set review for change "mobile~234239": HTTP status 409 Conflict; Change 234239 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:29 No reviewers or cc: https://go-review.googlesource.com/c/mobile/+/240009
2020/07/10 13:27:30 Setting review on https://go-review.googlesource.com/c/mobile/+/240009: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:hyangah@gmail.com State:}]}
2020/07/10 13:27:30 Could not set review for change "mobile~240009": HTTP status 409 Conflict; Change 240009 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:30 No reviewers or cc: https://go-review.googlesource.com/c/go/+/223118
2020/07/10 13:27:30 Setting review on https://go-review.googlesource.com/c/go/+/223118: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:bradfitz@golang.org State:}]}
2020/07/10 13:27:30 Could not set review for change "go~223118": HTTP status 409 Conflict; Change 223118 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:30 No reviewers or cc: https://go-review.googlesource.com/c/tools/+/223779
2020/07/10 13:27:30 Setting review on https://go-review.googlesource.com/c/tools/+/223779: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:dmitshur@golang.org State:}]}
2020/07/10 13:27:30 Could not set review for change "tools~223779": HTTP status 409 Conflict; Change 223779 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:30 No reviewers or cc: https://go-review.googlesource.com/c/tools/+/231464
2020/07/10 13:27:30 Setting review on https://go-review.googlesource.com/c/tools/+/231464: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:dmitshur@golang.org State:}]}
2020/07/10 13:27:30 Could not set review for change "tools~231464": HTTP status 409 Conflict; Change 231464 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:30 No reviewers or cc: https://go-review.googlesource.com/c/tools/+/239498
2020/07/10 13:27:31 Setting review on https://go-review.googlesource.com/c/tools/+/239498: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:matloob@golang.org State:}]}
2020/07/10 13:27:31 Could not set review for change "tools~239498": HTTP status 409 Conflict; Change 239498 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:31 No reviewers or cc: https://go-review.googlesource.com/c/net/+/187478
2020/07/10 13:27:31 Setting review on https://go-review.googlesource.com/c/net/+/187478: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:mikioh.public.networking@gmail.com State:}]}
2020/07/10 13:27:31 Could not set review for change "net~187478": HTTP status 409 Conflict; Change 187478 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.

It's not possible to leave review comments on those CLs.

Need to investigate why this is happening.

From initial observations, it might be CLs that GerritBot imported from PRs, where PRs have had large review comments posted (e.g., golang/tools#215 (review)), which might have caused some sort of fail-and-repeat loops to make an excess of 5000 API calls to the Gerrit CL, causing Gerrit to lock those CLs. But I've only looked very briefly so far, more investigation is needed to confirm this or find another explanation.

/cc @golang/osp-team
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 10, 2020
@dmitshur dmitshur added this to the Unreleased milestone Jul 10, 2020
@dmitshur
Copy link
Contributor Author

The relevant code for this error message in Gerrit source code is at https://gerrit.googlesource.com/gerrit/+/refs/tags/v3.2.2/java/com/google/gerrit/server/notedb/OpenRepo.java#181.

@dmitshur dmitshur mentioned this issue Aug 6, 2020
Closed
@andybons
Copy link
Member

andybons commented Aug 6, 2020

I’m going to temporarily disabled assigning reviewers to CLs, as it’s possible that there’s an infinite loop there.

@gopherbot
Copy link

Change https://golang.org/cl/247195 mentions this issue: cmd/gopherbot: temporarily disable assigning reviewers

gopherbot pushed a commit to golang/build that referenced this issue Aug 6, 2020
@andybons
cmd/gopherbot: temporarily disable assigning reviewers
c32acf8 
This is potentially caught in an infinite loop, which updates
a Gerrit change too many times, leaving the change in an unusable
state.

Also adds a map of deleted issues to prevent an error loop caused
by attempting to post changes to issues that don’t exist.

Updates golang/go#40147
Updates golang/go#40468

Change-Id: I3922d63c146591809324861d40437f43e115748c
Reviewed-on: https://go-review.googlesource.com/c/build/+/247195
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@dmitshur
Copy link
Contributor Author

We ran into this in https://go.dev/cl/231464 (created in May 2020), reported by @ianthehat recently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@andybons @dmitshur @gopherbot