Skip to content

x/build/cmd/gitmirror: Go repo mirroring hung #35124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dmitshur opened this issue Oct 23, 2019 · 5 comments
Closed

x/build/cmd/gitmirror: Go repo mirroring hung #35124

dmitshur opened this issue Oct 23, 2019 · 5 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

From https://farmer.golang.org/#health:

image

6 hours is abnormal.

The last Go commit on https://build.golang.org is currently f91e895 ("syscall: reenable sysctl on iOS"), same as on the GitHub repo mirror. So this is affecting post-submit builders too, not just GitHub repo mirroring.
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations) labels Oct 23, 2019
@gopherbot gopherbot added this to the Unreleased milestone Oct 23, 2019
@dmitshur dmitshur changed the title x/build: Go repo mirroring hung x/build/cmd/gitmirror: Go repo mirroring hung Oct 23, 2019
@dmitshur
Copy link
Contributor Author

Log from its /debug/watcher/go endpoint:


watcher status for repo: "go"
2019-10-23T16:22:49Z   6h34m55s ago         running git fetch origin
2019-10-23T16:22:49Z   6h34m55s ago         poll timer fired
2019-10-23T16:17:49Z   6h39m55s ago         waiting
2019-10-23T16:17:49Z   6h39m55s ago         updated dashboard
2019-10-23T16:17:49Z   6h39m55s ago         updating dashboard
2019-10-23T16:17:49Z   6h39m55s ago         did sync to github
2019-10-23T16:17:31Z   6h40m13s ago         syncing to github
2019-10-23T16:17:31Z   6h40m13s ago         ran git fetch
2019-10-23T16:17:17Z   6h40m26s ago         running git fetch origin
2019-10-23T16:17:17Z   6h40m26s ago         poll timer fired
2019-10-23T16:12:17Z   6h45m26s ago         waiting
2019-10-23T16:12:17Z   6h45m26s ago         updated dashboard
2019-10-23T16:12:17Z   6h45m27s ago         updating dashboard
2019-10-23T16:12:17Z   6h45m27s ago         did sync to github
2019-10-23T16:11:59Z   6h45m44s ago         syncing to github
2019-10-23T16:11:59Z   6h45m44s ago         ran git fetch
2019-10-23T16:11:44Z   6h46m0s ago          running git fetch origin
2019-10-23T16:11:44Z   6h46m0s ago          poll timer fired
2019-10-23T16:06:44Z   6h51m0s ago          waiting
2019-10-23T16:06:44Z   6h51m0s ago          updated dashboard
2019-10-23T16:06:44Z   6h51m0s ago          updating dashboard
2019-10-23T16:06:44Z   6h51m0s ago          did sync to github
2019-10-23T16:06:26Z   6h51m18s ago         syncing to github
2019-10-23T16:06:26Z   6h51m18s ago         ran git fetch
2019-10-23T16:06:08Z   6h51m36s ago         running git fetch origin
2019-10-23T16:06:08Z   6h51m36s ago         poll timer fired
2019-10-23T16:01:08Z   6h56m36s ago         waiting
2019-10-23T16:01:08Z   6h56m36s ago         updated dashboard
2019-10-23T16:01:08Z   6h56m36s ago         updating dashboard
2019-10-23T16:01:08Z   6h56m36s ago         did sync to github
2019-10-23T16:00:50Z   6h56m54s ago         syncing to github
2019-10-23T16:00:50Z   6h56m54s ago         ran git fetch
2019-10-23T16:00:35Z   6h57m9s ago          running git fetch origin
2019-10-23T16:00:35Z   6h57m9s ago          poll timer fired
2019-10-23T15:55:35Z   7h2m9s ago           waiting
2019-10-23T15:55:35Z   7h2m9s ago           updated dashboard
2019-10-23T15:55:34Z   7h2m9s ago           updating dashboard
2019-10-23T15:55:34Z   7h2m9s ago           did sync to github
2019-10-23T15:55:16Z   7h2m28s ago          syncing to github
2019-10-23T15:55:16Z   7h2m28s ago          ran git fetch
2019-10-23T15:54:57Z   7h2m47s ago          running git fetch origin
2019-10-23T15:54:57Z   7h2m47s ago          poll timer fired
2019-10-23T15:49:57Z   7h7m47s ago          waiting
2019-10-23T15:49:57Z   7h7m47s ago          updated dashboard
2019-10-23T15:49:57Z   7h7m47s ago          updating dashboard
2019-10-23T15:49:57Z   7h7m47s ago          did sync to github
2019-10-23T15:49:38Z   7h8m5s ago           syncing to github
2019-10-23T15:49:38Z   7h8m5s ago           ran git fetch
2019-10-23T15:49:22Z   7h8m21s ago          running git fetch origin

It seems to be stuck on the git fetch command execution:

https://github.com/golang/build/blob/39f3d49936c64caf850d1d712e4e2c7c781b332a/cmd/gitmirror/gitmirror.go#L1106

I'll add a generous timeout.

@dmitshur dmitshur self-assigned this Oct 23, 2019
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Oct 23, 2019
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/203057 mentions this issue: cmd/gitmirror: add 10 minute timeout to git fetch

@dmitshur
Copy link
Contributor Author

The immediate issue is resolved; the go repo is being mirrored and its commits are sent to post-submit builders. Removing "Soon" label. /cc @golang/osp-team

image

image

@dmitshur dmitshur removed the Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations) label Oct 23, 2019
@ALTree ALTree mentioned this issue Oct 24, 2019
Closed
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/205860 mentions this issue: maintner: add tracing on 'git fetch origin'

gopherbot pushed a commit to golang/build that referenced this issue Nov 7, 2019
maintner: add tracing on 'git fetch origin'
2923e3d 
Also increase its timeout and terminate it with a friendlier signal
and add some logging of how long git operations took.

Updates golang/go#35293
Updates golang/go#35124

Change-Id: I1ed466d872a11f60751953ef5274be96cea0294b
Reviewed-on: https://go-review.googlesource.com/c/build/+/205860
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@dmitshur @codebien
cmd/gitmirror: add 10 minute timeout to git fetch
6d6a10b 
The Repo.fetch method tries to execute "git fetch" up to 3 times with
linear back-off. We don't expect the git fetch process to take more
than a few minutes unless something is very wrong. At that point,
it's better to abort and try again. So, add a 10 minute timeout.

This should help prevent fetch from being stuck and never retrying.

Fixes golang/go#35124

Change-Id: Ic06a552378f8bff1bea314288a3637548663a3ef
Reviewed-on: https://go-review.googlesource.com/c/build/+/203057
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@codebien
maintner: add tracing on 'git fetch origin'
ccbb848 
Also increase its timeout and terminate it with a friendlier signal
and add some logging of how long git operations took.

Updates golang/go#35293
Updates golang/go#35124

Change-Id: I1ed466d872a11f60751953ef5274be96cea0294b
Reviewed-on: https://go-review.googlesource.com/c/build/+/205860
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/227858 mentions this issue: cmd/gitmirror: update build image to Go 1.14

gopherbot pushed a commit to golang/build that referenced this issue Apr 10, 2020
@dmitshur
cmd/gitmirror: update build image to Go 1.14
f87aadf 
The timeout added in CL 35124 doesn't seem to be working as intended.
Update to a supported version of Go before investigating further.

Also remove the explicit GOPROXY override, since the default behavior
as of Go 1.13 uses the Go module mirror¹.

¹ https://golang.org/doc/go1.13#introduction

For golang/go#35124.

Change-Id: Ia01abc530f0284791fe6e57ce5700c0ebc75268e
Reviewed-on: https://go-review.googlesource.com/c/build/+/227858
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@dmitshur dmitshur mentioned this issue May 5, 2020
Closed
@golang golang locked and limited conversation to collaborators Apr 10, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
2 participants
@dmitshur @gopherbot