Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compress/zlib: compressed data is different from zlib-flate #53232

Closed
aagu opened this issue Jun 4, 2022 · 2 comments
Closed

compress/zlib: compressed data is different from zlib-flate #53232

aagu opened this issue Jun 4, 2022 · 2 comments
Labels
FrozenDueToAge

Comments

@aagu
Copy link

aagu commented Jun 4, 2022

What version of Go are you using (go version)?

$ go version
go version go1.18.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/aagu/.cache/go-build"
GOENV="/home/aagu/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/aagu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/aagu/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.2"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build718718165=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I am learning about the git object storage recently. I tried to use the compress/zlib to implment the git hash-object command, the snippet of exmple code could be found at https://go.dev/play/p/_a_CGxRHTcy. The compress data will be write to the .git/objects/[ref-header]/[ref-body].

What did you expect to see?

The content of git object can be displayed as: git cat-file -p [some digest].

What did you see instead?

The content could not be read by git:

git cat-file -p e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
fatal: unable to stream e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 to stdout

The content compress by compress/zlib was:

cat .git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 | xxd
00000000: 785e 4aca c94f 5230 6000 0000 00ff ff    x^J..OR0`......

I tried to decompress and recompress the content like this:

cat .git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 | zlib-flate -uncompress | zlib-flate -compress=5 | xxd
00000000: 785e 4bca c94f 5230 6000 0009 b001 f0    x^K..OR0`......

The output is now be the same with git hash-object does.

I have tried with compress level from 1 to 9 of the compress/zlib lib, but none of them can produce 785e 4bca c94f 5230 6000 0009 b001 f0

I am not familiar with the zlib algo, can any one be kind to help me out? Thanks in advance!
@dsnet
Copy link
Member

dsnet commented Jun 4, 2022

The underlying compression format is RFC 1951. The Go implementation only guarantees that it produces valid output that is compatible with RFC 1951, but does not guarantee that it outputs any specific sequence of bits.

@aagu
Copy link
Author

aagu commented Jun 4, 2022

@dsnet Thanks for the reply, it turns out that I just called Flush instead of Close for the zlib writer, so the output write to file is not correct.

@aagu aagu closed this as completed Jun 4, 2022
@golang golang locked and limited conversation to collaborators Jun 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dsnet @aagu @gopherbot