Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/distpack: consider the use of zopfli #62445

Closed
dsnet opened this issue Sep 4, 2023 · 4 comments
Closed

cmd/distpack: consider the use of zopfli #62445

dsnet opened this issue Sep 4, 2023 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@dsnet
Copy link
Member

dsnet commented Sep 4, 2023 •
edited

There is benefit to providing releases compressed with gzip over formats like zstd or xz.
While the other formats provide better compression ratios, they are seldom used in comparison to the ubiquitous nature of gzip.
zstd is well-positioned to take over as the defacto compression format, but that probably won't happen for another decade.

That said, we can still do better within gzip itself.

If we're willing to burn CPU and depend on cgo,
we can use zopfli to produce a close-to-optimal gzip file.
Note that zopfli is not a new compression format, but a clever algorithm that searches for the optimal selection of LZ77 matches and block splits in the existing gzip format.

For example, with go1.21.0.linux-amd64.tar.gz, the current size is 63.4MiB.
But after 15 minutes of single-threaded execution on a Ryzan 5900x with zopfli,
the file can be reduced to 59.7MiB, which is a savings of ~6%.
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Sep 4, 2023
@gopherbot gopherbot added this to the Unreleased milestone Sep 4, 2023
@cespare cespare mentioned this issue Sep 4, 2023
Open
@cespare
Copy link
Contributor

cespare commented Sep 4, 2023

On one hand, this seems like a clear win. On the other, it seems ridiculous to participate in this herculean effort to push 90s tech to the utmost limit when much, much better options are available in 2023. I filed #62446 to register this view, though my hopes for uptake are not high.

@heschi heschi changed the title x/build/cmd/relui: consider the use of zopfli cmd/distpack: consider the use of zopfli Sep 5, 2023
@cherrymui
Copy link
Member

cc @rsc @golang/release

@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 5, 2023
@cherrymui cherrymui modified the milestones: Unreleased, Backlog Sep 5, 2023
@dsnet
Copy link
Member Author

dsnet commented Sep 5, 2023

For the record, I like @cespare's proposal in #62446. I'm a fan of zstandard. Providing zstd releases alongside gzip might obviate the need for this, although they're not mutually exclusive solutions. Burning 15 minutes of CPU would save 3.5TB of bandwidth if there was a 1M downloads.

@heschi
Copy link
Contributor

heschi commented Sep 19, 2023

I think if we do anything here it'll probably be to add zstd archives. Closing for now.

@heschi heschi closed this as not planned Won't fix, can't repro, duplicate, stale Sep 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Go Release
Archived in project
Milestone
Backlog
Development

No branches or pull requests
5 participants
@cespare @dsnet @gopherbot @cherrymui @heschi