Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/gitmirror: panic in parseRefs #29560

Closed
dmitshur opened this issue Jan 4, 2019 · 2 comments
Closed

x/build/cmd/gitmirror: panic in parseRefs #29560

dmitshur opened this issue Jan 4, 2019 · 2 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Jan 4, 2019 •
edited

Spotted the following crash due to a panic in gitmirror production logs. It happened once between December 13, 2018 and today, running version golang/build@a196f5a. Making an issue for future reference, but the urgency to fix it is not high.

[...]
2018/12/18 00:59:58 wrote /cache/golang-maintner/0044.growing.mutlog
2018/12/18 00:59:58 Reloaded data from log *maintner.netMutSource.
2018/12/18 00:59:58 Updating data from log *maintner.netMutSource ...
2018/12/18 00:59:58 Downloading 16535 bytes of https://maintner.golang.org/logs/44 ...
2018/12/18 00:59:58 wrote /cache/golang-maintner/0044.growing.mutlog
2018/12/18 00:59:58 Reloaded data from log *maintner.netMutSource.
2018/12/18 00:59:58 Updating data from log *maintner.netMutSource ...
2018/12/18 00:59:58 Downloading 517 bytes of https://maintner.golang.org/logs/44 ...
2018/12/18 00:59:58 wrote /cache/golang-maintner/0044.growing.mutlog
2018/12/18 00:59:59 gerrit code.googlesource.com/gocloud: Ref {CLNumber:36550 Version:0} => 9c82146fdb60bd3c20550b4338086d31be5308f6
2018/12/18 00:59:59 gerrit code.googlesource.com/gocloud: Ref {CLNumber:36270 Version:17} => 61be65d41596e538a13096fbfa3d54a0dfb5aaed
2018/12/18 00:59:59 gerrit code.googlesource.com/gocloud: Ref {CLNumber:36270 Version:0} => 818d367ffefe3962c5ce3514609e6eb09fa77b7e
2018/12/18 00:59:59 gerrit code.googlesource.com/gocloud: Ref {CLNumber:36590 Version:1} => 4dfd96393603a403058af13f23ce97c6b5216ebc
2018/12/18 00:59:59 gerrit code.googlesource.com/gocloud: Ref {CLNumber:36590 Version:0} => a20a43b1764b61da170128eb22c368033d724605
2018/12/18 00:59:59 Reloaded data from log *maintner.netMutSource.
2018/12/18 00:59:59 maintner refs for code.googlesource.com/gocloud changed
2018/12/18 00:59:59 Updating data from log *maintner.netMutSource ...
panic: runtime error: index out of range

goroutine 145 [running]:
main.parseRefs(0xc457e20580, 0xc420199bc0, 0x8ff693, 0x3)
        /go/src/golang.org/x/build/cmd/gitmirror/gitmirror.go:1533 +0x4c4
main.(*Repo).getRemoteRefs(0xc420475600, 0x8ff8ae, 0x4, 0x0, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gitmirror/gitmirror.go:1514 +0x14a
main.(*Repo).push.func2(0x0, 0x10)
        /go/src/golang.org/x/build/cmd/gitmirror/gitmirror.go:1118 +0x255
main.try(0x3, 0xc440503dd8, 0xc42024fe08, 0xc420475600)
        /go/src/golang.org/x/build/cmd/gitmirror/gitmirror.go:1273 +0x6b
main.(*Repo).push(0xc420475600, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gitmirror/gitmirror.go:1104 +0xbc
main.(*Repo).Loop(0xc420475600)
        /go/src/golang.org/x/build/cmd/gitmirror/gitmirror.go:576 +0x24f
main.runGitMirror.func1(0xc42047a4b0, 0xa, 0xc4203edbc0, 0x17, 0x0)
        /go/src/golang.org/x/build/cmd/gitmirror/gitmirror.go:216 +0x2ae
created by main.runGitMirror
        /go/src/golang.org/x/build/cmd/gitmirror/gitmirror.go:239 +0x5ee

The relevant line is x/build/cmd/gitmirror/gitmirror.go:1533.

To resolve this issue, the first step is to understand why and under what conditions it happens. Help is welcome.
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jan 4, 2019
@gopherbot gopherbot added this to the Unreleased milestone Jan 4, 2019
@gopherbot
Copy link

Change https://golang.org/cl/156341 mentions this issue: cmd/gitmirror: prevent a panic and log on bogus ref line in parseRefs

gopherbot pushed a commit to golang/build that referenced this issue Jan 5, 2019
@bradfitz
cmd/gitmirror: prevent a panic and log on bogus ref line in parseRefs
46298fd 
Updates golang/go#29560

Change-Id: I397b38af3dc421b5de6e6a799ad6b0e5ccab62f3
Reviewed-on: https://go-review.googlesource.com/c/156341
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@bradfitz bradfitz changed the title x/build/cmd/gitmirror: possible panic in parseRefs x/build/cmd/gitmirror: panic in parseRefs Jan 5, 2019
@gopherbot
Copy link

Change https://golang.org/cl/190897 mentions this issue: cmd/gitmirror: replace batched git push mirroring with a single git push

@dmitshur dmitshur self-assigned this Aug 20, 2019
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@dmitshur @codebien
cmd/gitmirror: replace batched git push mirroring with a single git push
da5480a 
In 2016, there were problems with git mirroring being slow and flaky,
as described in golang/go#16388. In order to resolve that problem,
CL 25110 added a custom mirroring implementation that effectively
split up the work done by a single git push -f --mirror dest command
into smaller batches of work. It was noted in the commit message and
updated documentation of CL 127315 that it may no longer be needed in
newer versions of git, and that the problem seemed to affect the HTTPS
transport, not so much the SSH transport.

By now, the following conditions are different:

• It's 2019.
• We're using a newer version of git.
• We're using the SSH transport (which may or may not be better than
  the HTTPS transport).
• We are mirroring only the refs/heads/* and refs/tags/* namespaces,
  not refs/changes/* which contains an order of magnitude more refs.

As a result, it should be possible to revert to a simple mirroring
implementation that uses a single git push -f --mirror dest command.
This removes complexity in our code, leveraging the git binary to do
more for us (at the expense of relinquishing tighter control and
ability to do things differently if we start encountering problems).
It resolves the issue of deleted refs not being deleted from mirrors
(golang/go#23099) without having to add even more complexity.

Debian 10 (buster) is the current stable release of Debian.
Start using it, since it comes with a newer version of git
(git version 2.20.1, compared to git version 2.11.0 in Debian 9),
which we want in hopes of git being better capable of pushing
a large number of refs without problems.

If we run into problems again, we might have to add some of the
complexity back, but hopefully that won't happen. Remove the old
and unused code since it's easy to get it back from VCS history.

Fixes golang/go#23099
Fixes golang/go#29560
Updates golang/go#16388

Change-Id: Id63419a5e4df5a87d7c9dc4be4e7e38a4849b3df
Reviewed-on: https://go-review.googlesource.com/c/build/+/190897
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@golang golang locked and limited conversation to collaborators Aug 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
2 participants
@dmitshur @gopherbot