Skip to content

compress/flate: add support for deflate64 compression #33023

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
alex opened this issue Jul 10, 2019 · 6 comments
Closed

compress/flate: add support for deflate64 compression #33023

alex opened this issue Jul 10, 2019 · 6 comments
Labels
FeatureRequest Issues asking for a new feature that does not need a proposal. FrozenDueToAge NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.

Comments

@alex
Copy link
Contributor

alex commented Jul 10, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.5 linux/amd64

Does this issue reproduce with the latest release?

Yup

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/alex/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/alex/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="clang"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build508250989=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Attempt to use archive/zip.NewReader() to open a ZIP archive containing files compressed with deflate64.

What did you expect to see?

The archive is opened as seamlessly as if files were compressed with standard deflate.

What did you see instead?

zip: unsupported compression algorithm

I should note that deflate64 seems to be exceptionally poorly supported in the open source world, as a result of the scarcity of documentation from PKWARE (the originators of this algorithm), so I'd be understanding if Go also was not interested in supporting it. With that said, I'd find it exceptionally convenient if Go had native support for deflate64.

Wikipedia has a brief description of how deflate64 differs from deflate: https://en.wikipedia.org/wiki/DEFLATE#Deflate64/Enhanced_Deflate
@ianlancetaylor
Copy link
Member

CC @dsnet

@alex
Copy link
Contributor Author

alex commented Jul 10, 2019

To amend my comment on the scarcity of open source implementations, it turns out zlib contains an implementation in their contrib directory: https://github.com/madler/zlib/blob/master/contrib/infback9/

@dsnet
Copy link
Member

dsnet commented Jul 10, 2019

This is issue actually less about archive/zip and more about compress/flate since most of the logic would be there to support this.

However, I'm inclined to say that compress/flate should not support Deflate64 since it's a proprietary extension of RFC1951, and to my knowledge, not well-specified.

Given the fact that archive/zip already provides the means to register your own custom compressors, what's wrong with using a third party implementation of Deflate64 outside of standard library?

@alex
Copy link
Contributor Author

alex commented Jul 10, 2019 •
edited
Loading

You're right, I should have filed this against compress/flate.

Yes, it could absolutely live outside of the standard library, unfortunately right now there is no deflate64 implementation for Go, so it's not simply a matter of wiring one up to archive/zip. My argument for having it in the standard library are: a) the possibility that it could share code with compress/flate, b) it'd give archive/zip broader compatibility the standard tools like unzip(1), c) and this is the least persuasive -- the underspecification of the algorithm puts a premium on rigorous testing and other techniques for having confidence, being in the standard library is generally a good marker of such quality.

@dsnet
Copy link
Member

dsnet commented Jul 10, 2019

To be honest, I'm not sure deflate64 meets the bar for inclusion the standard library. The shared code argument alone is not sufficient. See https://golang.org/doc/faq#x_in_std

@dsnet dsnet changed the title archive/zip: Add support for deflate64 compression compress/flate: add support for deflate64 compression Jul 10, 2019
@dsnet dsnet added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Jul 10, 2019
@dsnet dsnet added the FeatureRequest Issues asking for a new feature that does not need a proposal. label Aug 3, 2019
@ALTree
Copy link
Member

ALTree commented Jan 28, 2020 •
edited
Loading

It seems like there isn't a strong argument for deflate64 inclusion in the standard library, considering it's 1) proprietary and underspecified 2) not commonly used; also the owner of the flate package seems unconvinced, and no one else has spoken up, so I think we can close this "needsDecision" issue as WONTFIX.

@ALTree ALTree closed this as completed Jan 28, 2020
@golang golang locked and limited conversation to collaborators Jan 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FeatureRequest Issues asking for a new feature that does not need a proposal. FrozenDueToAge NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@alex @ianlancetaylor @ALTree @dsnet @gopherbot