Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/gitmirror: deleted branches/tags are being mirrored to GitHub #23099

Closed
andybons opened this issue Dec 12, 2017 · 7 comments
Closed
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@andybons
Copy link
Member

I accidentally added some superfluous tags while doing the release last week. I removed them and they no longer show up on https://go.googlesource.com/go/+refs

On GitHub, however, they do show up even after I’ve manually deleted them in the GitHub UI. An example: https://github.com/golang/go/releases/tag/andybons%2Fcachedir.mailed

/cc @bradfitz

@gopherbot gopherbot added this to the Unreleased milestone Dec 12, 2017
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Dec 12, 2017
@dmitshur dmitshur changed the title x/build/cmd/gitmirror: deleted tags are being mirrored to GitHub x/build/cmd/gitmirror: deleted branches/tags are being mirrored to GitHub Jan 2, 2019
@dmitshur
Copy link
Contributor

dmitshur commented Jan 2, 2019

This is happening for deleted branches too. E.g., release-branch.go1.11-security was deleted from https://go.googlesource.com/go/+refs, but it's present in the GitHub mirror, even after being manually deleted one time from the GitHub mirror.

@dmitshur
Copy link
Contributor

Has something changed here? Both the tag mentioned in the original issue description and the security branches are not currently showing up on GitHub. Is the issue resolved, or is there another explanation?

@FiloSottile
Copy link
Contributor

I manually removed the -security branch, and it looks like it's not coming back.

@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 25, 2019
@dmitshur
Copy link
Contributor

dmitshur commented Aug 19, 2019

While it may have looked like something changed causing this issue to not be a problem anymore, that is not the case.

I've recently created and later deleted a branch named dmitshur-test1 in the scratch subrepo. It does not exist on the origin server anymore. However, it didn't get deleted from the GitHub mirror. When I deleted it manually, it re-appeared. I got the following from gitmirror logs:

[...]
2019/08/19 19:06:33 Reloaded data from log *maintner.netMutSource.
2019/08/19 19:06:33 Updating data from log *maintner.netMutSource ...
2019/08/19 19:06:34 scratch: 1 refs remain to sync to github
remote: 
remote: Create a pull request for 'dmitshur-test1' on GitHub by visiting:        
remote:      https://github.com/golang/scratch/pull/new/dmitshur-test1        
remote: 
To github.com:golang/scratch.git
 * [new branch]      1d5b30010bae27fa3b85e03370c17c837c095fa1 -> dmitshur-test1
2019/08/19 19:07:42 Downloading 33 bytes of https://maintner.golang.org/logs/54 ...
2019/08/19 19:07:42 wrote /cache/golang-maintner/0054.growing.mutlog
2019/08/19 19:07:42 Reloaded data from log *maintner.netMutSource.
[...]

Possible Explanation

I have a theory about what's causing this issue and why it seems to get resolved on its own after some time (but then reoccurs again).

Perhaps the way the origin repository is cloned to a temporary location in the gitmirror worker is such that it doesn't track branch/tag deletions from upstream, so gitmirror keeps thinking the origin has them and keeps pushing them to the GitHub mirror. However, maybe after gitmirror is redeployed it creates a fresh clone of the upstream repositories that no longer has the branches/tags deleted from upstream, and so the problem stops happening.

If that's the case, the fix will be to change gitmirror so that it tracks branch/tag removals from the origin repository. I think it can be done by using --prune flag, e.g., git remote update --prune or git fetch --prune origin (see https://git-scm.com/docs/git-remote and https://git-scm.com/docs/git-fetch#_pruning). Also, this line might be relevant.

@gopherbot
Copy link

Change https://golang.org/cl/190879 mentions this issue: cmd/gitmirror: prune refs that no longer exist on origin

@gopherbot
Copy link

Change https://golang.org/cl/190897 mentions this issue: cmd/gitmirror: try using git push instead of custom implementation

gopherbot pushed a commit to golang/build that referenced this issue Aug 20, 2019
Use the --prune flag when running git fetch origin commands
to delete remote-tracking refs in the local repositories.
This prevents branches and tags that have been deleted on
the origin server from reappearing in the GitHub mirrors.

It will take more work to get refs automatically deleted from
the mirror repositories, because the current Repo.push uses
a custom implementation rather than git push -f --mirror dest,
and it doesn't take the need to delete refs into account yet.

See https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---prune
for information on the --prune flag. The --prune-tags flag isn't
needed because the local repositories are cloned with --mirror.

Updates golang/go#23099

Change-Id: I762b2d1365287d5dac40236310713c3df15d59d8
Reviewed-on: https://go-review.googlesource.com/c/build/+/190879
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
@dmitshur
Copy link
Contributor

I've investigated this further, it seems the theory above was accurate.

CL 190879 is deployed and makes it so that refs deleted from origin repositories are no longer continuously recreated on the GitHub mirrors.

They're not automatically deleted yet, but with a little more work (CL 190897 is underway) that should be possible to resolve too.

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Aug 20, 2019
@dmitshur dmitshur self-assigned this Aug 20, 2019
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
Use the --prune flag when running git fetch origin commands
to delete remote-tracking refs in the local repositories.
This prevents branches and tags that have been deleted on
the origin server from reappearing in the GitHub mirrors.

It will take more work to get refs automatically deleted from
the mirror repositories, because the current Repo.push uses
a custom implementation rather than git push -f --mirror dest,
and it doesn't take the need to delete refs into account yet.

See https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---prune
for information on the --prune flag. The --prune-tags flag isn't
needed because the local repositories are cloned with --mirror.

Updates golang/go#23099

Change-Id: I762b2d1365287d5dac40236310713c3df15d59d8
Reviewed-on: https://go-review.googlesource.com/c/build/+/190879
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
In 2016, there were problems with git mirroring being slow and flaky,
as described in golang/go#16388. In order to resolve that problem,
CL 25110 added a custom mirroring implementation that effectively
split up the work done by a single git push -f --mirror dest command
into smaller batches of work. It was noted in the commit message and
updated documentation of CL 127315 that it may no longer be needed in
newer versions of git, and that the problem seemed to affect the HTTPS
transport, not so much the SSH transport.

By now, the following conditions are different:

• It's 2019.
• We're using a newer version of git.
• We're using the SSH transport (which may or may not be better than
  the HTTPS transport).
• We are mirroring only the refs/heads/* and refs/tags/* namespaces,
  not refs/changes/* which contains an order of magnitude more refs.

As a result, it should be possible to revert to a simple mirroring
implementation that uses a single git push -f --mirror dest command.
This removes complexity in our code, leveraging the git binary to do
more for us (at the expense of relinquishing tighter control and
ability to do things differently if we start encountering problems).
It resolves the issue of deleted refs not being deleted from mirrors
(golang/go#23099) without having to add even more complexity.

Debian 10 (buster) is the current stable release of Debian.
Start using it, since it comes with a newer version of git
(git version 2.20.1, compared to git version 2.11.0 in Debian 9),
which we want in hopes of git being better capable of pushing
a large number of refs without problems.

If we run into problems again, we might have to add some of the
complexity back, but hopefully that won't happen. Remove the old
and unused code since it's easy to get it back from VCS history.

Fixes golang/go#23099
Fixes golang/go#29560
Updates golang/go#16388

Change-Id: Id63419a5e4df5a87d7c9dc4be4e7e38a4849b3df
Reviewed-on: https://go-review.googlesource.com/c/build/+/190897
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@golang golang locked and limited conversation to collaborators Aug 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

4 participants