Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/gitmirror: service problem on 2024-04-23 #67070

Closed
dmitshur opened this issue Apr 26, 2024 · 1 comment
Closed

x/build/cmd/gitmirror: service problem on 2024-04-23 #67070

dmitshur opened this issue Apr 26, 2024 · 1 comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

Reporting for posterity and in case this comes up again. https://farmer.golang.org/#health was reporting a problem with Git mirroring on 2024-04-23:

image

It was unhealthy and continuously restarting:

$ kubectl get pods
NAME                                              READY   STATUS    RESTARTS          AGE
[...]
gitmirror-mirroring-deployment-6ffdbb4bbb-wtsfs   1/1     Running   170 (4m40s ago)   28d
gitmirror-serving-deployment-76944dc76b-h2qpv     1/1     Running   1 (15h ago)       28d
gitmirror-serving-deployment-76944dc76b-lmk8r     1/1     Running   1 (15h ago)       28d
[...]

The specific problem it kept running into was a "invalid index-pack output" error while cloning the go repository:

2024/04/23 14:31:55 go: git [clone --mirror https://go.googlesource.com/go /cache/gitmirror/go] failed after 5m33.999593098s: exit status 128
stderr: Cloning into bare repository '/cache/gitmirror/go'...
fatal: fetch-pack: invalid index-pack output

2024/04/23 14:31:55 initializing repos: cloning https://go.googlesource.com/go: exit status 128

CC @golang/release, @joedian.
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. labels Apr 26, 2024
@dmitshur dmitshur added this to the Unreleased milestone Apr 26, 2024
@dmitshur
Copy link
Contributor Author

That error doesn't provide detail but suggests an internal problem in git, possibly related to insufficient resources. We hadn't increased gitmirror's resources in a while.

That said, after redeploying the latest version, the problem hasn't returned, and it's been working okay:

2024/04/23 14:53:15 go: ran git [clone --mirror https://go.googlesource.com/go /cache/gitmirror/go] in 7m26.809263865s

Closing since this instance is resolved. If it keeps coming up, we may need to do more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release
Status: Done
Milestone
Unreleased
Development

No branches or pull requests
1 participant
@dmitshur