Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/gitmirror: smarter git exporter filtering #11224

Closed
bradfitz opened this issue Jun 15, 2015 · 4 comments
Closed

x/build/cmd/gitmirror: smarter git exporter filtering #11224

bradfitz opened this issue Jun 15, 2015 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted
Milestone
Unplanned

Comments

@bradfitz
Copy link
Contributor

Add the capability for the watcher's git archive server to optionally filter out content.

Then when the coordinator is asking for the "talks" repo, it can skip the 30+ MB of binaries.

We already do this with the .gitattributes file (see https://golang.org/cl/11050 for #11199) but that's a hack. We should revert that once the coordinator knows how to ask the watcher to filter stuff.

/cc @adg @minux
@bradfitz bradfitz added the Builders x/build issues (builders, bots, dashboards) label Jun 15, 2015
@bradfitz bradfitz added this to the Unplanned milestone Jun 15, 2015
@bradfitz bradfitz changed the title x/build: smarter git exporter in the watcher x/build/cmd/gitmirror: smarter git exporter in the watcher Feb 11, 2017
@bradfitz bradfitz changed the title x/build/cmd/gitmirror: smarter git exporter in the watcher x/build/cmd/gitmirror: smarter git exporter filtering Feb 17, 2017
@bradfitz bradfitz self-assigned this Feb 17, 2017
@kevinburke
Copy link
Contributor

I can take this one.

@kevinburke
Copy link
Contributor

Thoughts on how the coordinator should ask to exclude files?

GET /build.tar.gz?exclude=foo.jpg&excludeGlob=*.png

@kevinburke
Copy link
Contributor

This might be a little tricky. If I am reading correctly the only way to exclude files from git archive is via a .gitattributes file. We could write this file based on the "ignore" values of the incoming request, but if I am reading correctly git archive hardcodes the .gitattributes filename, so we can only have one incoming request served at a time and that request has to clobber the .gitattributes file - you can't do like git archive --with-attributes=/tmp/foo/.gitattributes.

Another option would be to write the gzip file, open it with a gzip.Reader, and strip files out that we want to exclude before sending it over the wire.

https://git-scm.com/docs/git-archive

@bradfitz bradfitz mentioned this issue Feb 18, 2017
Closed
@bradfitz
Copy link
Contributor Author

Closing this. Not worth the effort. We can reopen later if the need arises.

@golang golang locked and limited conversation to collaborators May 29, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
3 participants
@bradfitz @kevinburke @gopherbot