Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

archive/zip: reading many small files allocates a lot of memory #14289

Closed
dziemba opened this issue Feb 10, 2016 · 3 comments
Closed

archive/zip: reading many small files allocates a lot of memory #14289

dziemba opened this issue Feb 10, 2016 · 3 comments
Labels
FrozenDueToAge
Milestone
Go1.7

Comments

@dziemba
Copy link
Contributor

dziemba commented Feb 10, 2016

archive/zip creates a new compress/flate.Reader for each file read from the archive. If there are many small files in an archive, this allocates a lot of memory.

compress/flate allows re-using of Readers, see https://codereview.appspot.com/97140043/

Using this in archive/zip reduces the amount of memory allocated/GCed. In a real-world scenario when reading ~ 500 small files from a ZIP archive this gives a speedup of 1.5x-2x.

Version: go1.6rc2
@ianlancetaylor ianlancetaylor added this to the Go1.7 milestone Feb 10, 2016
@dziemba
Copy link
Contributor Author

dziemba commented Feb 10, 2016

This should fix it: https://go-review.googlesource.com/#/c/19416/

But I'm not 100% sure if the implementaion is correct. Please review with care ;)

@gopherbot
Copy link

CL https://golang.org/cl/19416 mentions this issue.

@dziemba
Copy link
Contributor Author

dziemba commented May 4, 2016

Can we merge this soon so it will be included in Go 1.7?

@golang golang locked and limited conversation to collaborators May 4, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.7
Development

No branches or pull requests
3 participants
@ianlancetaylor @dziemba @gopherbot