Skip to content

x/build/cmd/gitmirror: service problem first detected on 2024-04-23 #67070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dmitshur opened this issue Apr 26, 2024 · 4 comments
Closed

x/build/cmd/gitmirror: service problem first detected on 2024-04-23 #67070

dmitshur opened this issue Apr 26, 2024 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

Reporting for posterity and in case this comes up again. https://farmer.golang.org/#health was reporting a problem with Git mirroring on 2024-04-23:

image

It was unhealthy and continuously restarting:

$ kubectl get pods
NAME                                              READY   STATUS    RESTARTS          AGE
[...]
gitmirror-mirroring-deployment-6ffdbb4bbb-wtsfs   1/1     Running   170 (4m40s ago)   28d
gitmirror-serving-deployment-76944dc76b-h2qpv     1/1     Running   1 (15h ago)       28d
gitmirror-serving-deployment-76944dc76b-lmk8r     1/1     Running   1 (15h ago)       28d
[...]

The specific problem it kept running into was a "invalid index-pack output" error while cloning the go repository:

2024/04/23 14:31:55 go: git [clone --mirror https://go.googlesource.com/go /cache/gitmirror/go] failed after 5m33.999593098s: exit status 128
stderr: Cloning into bare repository '/cache/gitmirror/go'...
fatal: fetch-pack: invalid index-pack output

2024/04/23 14:31:55 initializing repos: cloning https://go.googlesource.com/go: exit status 128

CC @golang/release, @joedian.
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. labels Apr 26, 2024
@dmitshur dmitshur added this to the Unreleased milestone Apr 26, 2024
@dmitshur dmitshur moved this to In Progress in Go Release Apr 26, 2024
@dmitshur
Copy link
Contributor Author

That error doesn't provide detail but suggests an internal problem in git, possibly related to insufficient resources. We hadn't increased gitmirror's resources in a while.

That said, after redeploying the latest version, the problem hasn't returned, and it's been working okay:

2024/04/23 14:53:15 go: ran git [clone --mirror https://go.googlesource.com/go /cache/gitmirror/go] in 7m26.809263865s

Closing since this instance is resolved. If it keeps coming up, we may need to do more.

@github-project-automation github-project-automation bot moved this from In Progress to Done in Go Release Apr 26, 2024
@cagedmantis cagedmantis reopened this May 20, 2024
@dmitshur dmitshur moved this from Done to In Progress in Go Release May 20, 2024
@cagedmantis
Copy link
Contributor

This error has happend again. I noticed that the memory limits were being exceeded for individual pods. the CPU limits were also very close to being exceeded. Increasing the allocated resources seems to have fixed it. I will continue to monitor the service until I feel confident that the issue is resolved.

@dmitshur dmitshur changed the title x/build/cmd/gitmirror: service problem on 2024-04-23 x/build/cmd/gitmirror: service problem first detected on 2024-04-23 May 20, 2024
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/586778 mentions this issue: cmd/gitmirror: increase CPU and memory resources

gopherbot pushed a commit to golang/build that referenced this issue May 20, 2024
@cagedmantis
cmd/gitmirror: increase CPU and memory resources
Loading
Loading status checks…
75ff9ef 
This increases the CPU and memory resources for the git-mirror serving
and mirror deployments. The utilized resources were reaching the allocated
limits.

For golang/go#67070

Change-Id: I646d9feab0385fb0ea1f21d25a2c25ffbafb2f5b
Reviewed-on: https://go-review.googlesource.com/c/build/+/586778
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
@cagedmantis
Copy link
Contributor

CL https://go.dev/cl/586778 seems to have fixed this issue.

@github-project-automation github-project-automation bot moved this from In Progress to Done in Go Release May 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release
Archived in project
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@cagedmantis @dmitshur @gopherbot