Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/gitmirror: no need to keep entire corpus in memory #35977

Closed
bradfitz opened this issue Dec 4, 2019 · 2 comments
Closed

x/build/cmd/gitmirror: no need to keep entire corpus in memory #35977

bradfitz opened this issue Dec 4, 2019 · 2 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. Performance
Milestone

Comments

@bradfitz
Copy link
Contributor

bradfitz commented Dec 4, 2019

The gitmirror process keeps the the full maintner corpus in memory to see when refs change and force a sync (otherwise the sync happens every 5 minutes).

To reduce its memory requirements and make it start faster, it could instead just watch the stream of maintner events and use that to detect ref changes, rather than using godata to slurp the entire project history into memory.

@gopherbot gopherbot added this to the Unreleased milestone Dec 4, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Dec 4, 2019
@gopherbot
Copy link

Change https://golang.org/cl/209962 mentions this issue: maintner: add TailNetworkMutationSource and new cmd/maintwatch command

gopherbot pushed a commit to golang/build that referenced this issue Dec 5, 2019
For debugging maintner in production, curiosity, and to optimize
gitmirror (golang/go#35977).

Updates golang/go#35977

Change-Id: Ie06d2b3f36d03eea613e60a7e60c14b2c45c89a4
Reviewed-on: https://go-review.googlesource.com/c/build/+/209962
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@toothrot toothrot added the NeedsFix The path to resolution is known, but the work has not been done. label Dec 5, 2019
@gopherbot
Copy link

Change https://golang.org/cl/210277 mentions this issue: cmd/gitmirror: don't keep entire maintner corpus in memory

@golang golang locked and limited conversation to collaborators Dec 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. Performance
Projects
None yet
Development

No branches or pull requests

3 participants