Skip to content

x/build/cmd/coordinator: exceeding Gerrit tree fetch limit should be a fatal error #49707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dmitshur opened this issue Nov 21, 2021 · 1 comment
Closed

x/build/cmd/coordinator: exceeding Gerrit tree fetch limit should be a fatal error #49707

dmitshur opened this issue Nov 21, 2021 · 1 comment
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Nov 21, 2021 •
edited
Loading

If a CL pushes the (compressed) size of the tree over a limit, trybots fail with something like:

  builder: linux-arm-aws
      rev: ac0da79a676c852fee4939b79baf97224cd0b334
 buildlet: https://18.219.134.19 EC2 VM: buildlet-linux-arm-aws-rnab13b9c
  started: 2021-11-21 15:57:17.181391097 +0000 UTC m=+162403.663911665
    ended: 2021-11-21 15:58:39.041888414 +0000 UTC m=+162485.524408982
  success: false

Events:
  2021-11-21T15:57:17Z checking_for_snapshot 
  2021-11-21T15:57:17Z finish_checking_for_snapshot after 17ms
  2021-11-21T15:57:17Z get_buildlet 
  2021-11-21T15:58:12Z finish_get_buildlet after 55.1s
  2021-11-21T15:58:12Z using_buildlet 18.219.134.19:443
  2021-11-21T15:58:12Z write_snapshot_tar 
  2021-11-21T15:58:18Z finish_write_snapshot_tar after 5.66s
  2021-11-21T15:58:18Z make_and_test 
  2021-11-21T15:58:18Z fetching_subrepo website
  2021-11-21T15:58:18Z get_source website@b4aa6ae539a1f644d48c473ca6e0e31a95ab4b26
  2021-11-21T15:58:38Z finish_get_source after 20.4s; err=reading website/b4aa6ae539a1f644d48c473ca6e0e31a95ab4b26 from gerrit: rejected because body exceeded a limit of 100 MB; see golang.org/issue/46379; website@b4aa6ae539a1f644d48c473ca6e0e31a95ab4b26
  2021-11-21T15:58:39Z finish_make_and_test after 20.5s; err=runTests: reading website/b4aa6ae539a1f644d48c473ca6e0e31a95ab4b26 from gerrit: rejected because body exceeded a limit of 100 MB; see golang.org/issue/46379

Build log:
linux-arm-aws at ac0da79a676c852fee4939b79baf97224cd0b334 building website at b4aa6ae539a1f644d48c473ca6e0e31a95ab4b26



Error: runTests: reading website/b4aa6ae539a1f644d48c473ca6e0e31a95ab4b26 from gerrit: rejected because body exceeded a limit of 100 MB; see golang.org/issue/46379

(From https://go-review.googlesource.com/c/website/+/365135/2//COMMIT_MSG#14.)

But it's currently considered a retryable problem, and since #13026 hasn't been resolved yet, it keeps on retrying forever until someone removes the Run-TryBot request.

Even if the issue above is fixed, coordinator should classify this as a non-retryable error and not keep retrying.
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 21, 2021
@dmitshur dmitshur added this to the Unreleased milestone Nov 21, 2021
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 21, 2021
@dmitshur dmitshur self-assigned this Nov 21, 2021
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/365974 mentions this issue: cmd/coordinator, internal/sourcecache: don't retry when tree too big

@heschi heschi moved this to Done in Go Release Sep 27, 2022
@golang golang locked and limited conversation to collaborators Jun 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release
Archived in project
Milestone
Unreleased
Development

No branches or pull requests
2 participants
@dmitshur @gopherbot