-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
compress/zlib: error due to compress/flate #11033
Comments
I have analyzed this data segment and confirmed that it is a duplicate of the issue #11030. For brevity, I isolated the raw DEFLATE stream: http://play.golang.org/p/KFEHgR2zal I decomposed the entire DEFLATE stream into the following (LSB on right): [
# Last, dynamic block
"1", "10",
# HLIT: 257, HDIST: 1, HCLEN: 12
"00000", "00000", "1000",
# HCLEN codes
"000", "011", "011", "010", "000", "000", "000", "011", "000", "010", "000", "011",
# HLIT tree
"10", "011", "001", "101", "00", "00", "101", "111", "0000101", "10", "011", "011", "10", "111", "0000010", "101", "00", "001", "10", "00", "00", "001", "10", "10", "111", "0001000", "10", "111", "0000001", "10", "011", "010", "001", "00", "10", "011", "010", "10", "00", "00", "001", "011", "100", "001", "111", "0000010", "10", "111", "0000110", "10", "00", "10", "111", "0101000", "001", "011", "110", "10", "011", "011", "101", "00", "101", "011", "110", "101", "011", "100", "101", "111", "0001000", "101", "001",
# HDIST tree
"00",
# Compressed data
"10001", "0000", "0000", "11001", "1000", "1000", "00001", "1100", "11101", "010111", "0010", "1100", "0010", "01110", "0100", "000111", "01001", "01101", "00011", "001111", "10101", "111111", "100111", "00101", "10011", "01011", "0100", "11110", "101111", "011111", "1010", "11011", "1010", "110111", "0110"
] Parsing the HCLEN codes, we generate the following Huffman table (LSB on right): 00 => 0
10 => 5
001 => 4
101 => 6
011 => 17
111 => 18 Thus, we can see that the HDIST tree is composed of a single zero-bit length (since the '00' code maps to the 0 symbol). To further confirm this, we can see the HDIST tree starts a bit offset 280. Converting to bytes, this is byte 35, which lies before the 36byte offset mentioned in the error. EDIT (further analysis): As a test, I modified the HDIST tree to be full tree. The only changes were changing the number of symbols in the HDIST tree (from 1 to 16) and changing the HDIST tree itself to contain 16 symbols, each of 4bits in length. This was the only way to generate a full HDIST tree without modifying the HCLENs. [
# Last, dynamic block
"1", "10",
# HLIT: 257, HDIST: 16 (CHANGED), HCLEN: 12
"00000", "01111", "1000",
# HCLEN codes
"000", "011", "011", "010", "000", "000", "000", "011", "000", "010", "000", "011",
# HLIT tree
"10", "011", "001", "101", "00", "00", "101", "111", "0000101", "10", "011", "011", "10", "111", "0000010", "101", "00", "001", "10", "00", "00", "001", "10", "10", "111", "0001000", "10", "111", "0000001", "10", "011", "010", "001", "00", "10", "011", "010", "10", "00", "00", "001", "011", "100", "001", "111", "0000010", "10", "111", "0000110", "10", "00", "10", "111", "0101000", "001", "011", "110", "10", "011", "011", "101", "00", "101", "011", "110", "101", "011", "100", "101", "111", "0001000", "101", "001",
# HDIST tree (CHANGED)
"001", "001", "001", "001", "001", "001", "001", "001", "001", "001", "001", "001", "001", "001", "001", "001",
# Compressed data
"10001", "0000", "0000", "11001", "1000", "1000", "00001", "1100", "11101", "010111", "0010", "1100", "0010", "01110", "0100", "000111", "01001", "01101", "00011", "001111", "10101", "111111", "100111", "00101", "10011", "01011", "0100", "11110", "101111", "011111", "1010", "11011", "1010", "110111", "0110"
] When composed into a byte stream, it now properly gets decoded by Go's flate library, further indicating that it is the HDIST tree causing issues. |
Closing as duplicate. |
Probably related to #11030.
Running this produces an error:
Playground link
I'm using
compress/zlib
heavily to process git packfiles and this error happens ocasionally.Versions tested:
go version go1.3.3 windows/amd64
go version go1.4.2 windows/amd64
go version go1.4.2 linux/amd64
go version go1.4.2 darwin/amd64
It's implementation related because Python (example is py3) does just fine:
The text was updated successfully, but these errors were encountered: