Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compress/bzip2: oversized block allocation #13941

Closed
mikispag opened this issue Jan 13, 2016 · 4 comments
Closed

compress/bzip2: oversized block allocation #13941

mikispag opened this issue Jan 13, 2016 · 4 comments
Labels
FrozenDueToAge
Milestone
Go1.7

Comments

@mikispag
Copy link

At line 79 of bzip2.go, you multiply level by 100 * 1024, while the original bzlib.c uses 1000 consistently. They also specify this in the documentation about allocation - see section 3.3.1 of http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html:

Parameter blockSize100k specifies the block size to be used for compression.
It should be a value between 1 and 9 inclusive, and the actual block size used
is 100000 x this figure. 9 gives the best compression but takes most memory.

So, instead of:

bz2.blockSize = 100 * 1024 * (int(level) - '0')

you should probably do:

bz2.blockSize = 100 * 1000 * (int(level) - '0')
@dsnet
Copy link
Member

dsnet commented Jan 13, 2016

You are correct that the block size used in compress/bzip2 is incorrect, but using a larger block size for decoding shouldn't cause any problems. It may cause it to be able to decode something that the C version otherwise would not be able to, but should not cause the Go implementation to fail on decoding something that the C version can.

Is this causing a problem for you?

The wrong block-size is not the only inconsistency between the Go version and the "canonical" C implementation.

@mikispag
Copy link
Author

You are right, this is not causing problems, but it is using more memory than necessary, and the fix is really easy!

@mdempsky mdempsky changed the title compress/bzip2: compress/bzip2: oversized block allocation Jan 13, 2016
@mdempsky mdempsky added this to the Go1.7 milestone Jan 13, 2016
@gopherbot
Copy link

CL https://golang.org/cl/20011 mentions this issue.

@mikispag
Copy link
Author

Good point - "the bzip2 decoder would incorrectly decode files with larger block sizes when it should have otherwise failed".

Thanks for fixing this!

@golang golang locked and limited conversation to collaborators Feb 28, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.7
Development

No branches or pull requests
4 participants
@mdempsky @mikispag @dsnet @gopherbot