cc @dsnet

Compression ratios are heavily dependent on the input data set. Without a reproducer, there isn't much that can be done here.

A few years ago, @klauspost optimized the encoder. As part of the optimization, it made the assumption that matches were always at least 4 bytes long. This means that we don't perform 3-byte matches, which may be the cause of lower compression ratio. See #15622. However, we cannot verify this without a reproduction.

If you want the same compression rate as another Lempel-Ziv compression tool, you need to replicate the Lempel-Ziv-Encoder exactly and use the same parameters. So this is actually a request to replicate the zlib Lempel-Ziv-Encoder in Go.

1. A sample size of one says nothing about a compression scheme.

Test on a large set of inputs. Your input clearly has some special properties since it is uncommon for higher levels to have worse compression than medium. This is not typical.

2. Size cannot be compared without considering speed.

For example, this is 4 GB web content speed vs compression with Go stdlib:

3. Compression levels should not be compared across implementations. They should reflect the tradeoff between speed an compression as an average in a given implementation. Looking at the graph above anything above level 6 looks like a waste of CPU cycles, and level 4 would be the reasonable "default", and not level 6.

For my own deflate I have tried to make the tradeoff more "linear", with a reasonable default level 5.

It is not perfect, and some inputs will skew it, but that is the measure I use for balancing the compression levels.

Typically I only really find the "fastest", "default" (the reasonable tradeoff) and "best" (smallest) levels to be useful anyway, so for most implementations that are the only ones I focus on.

Short story: Choose the compression level that makes sense for your use case. Don't copy+paste from other languages.

@dsnet 3 vs 4 byte is rarely a gain since encoding 3 byte matches in most cases yield the same or larger output than the entropy encoding of the immediate values. You can estimate both, but it is rather expensive and relies on incomplete information.

FWIW, I tried adding 3 byte matches to deflate level 9, without comparing entropy coded sizes similar to old implementation. Total output size went from 730389060 -> 732262492 bytes on webdevdata.org-2015-01-07-subset.
enwik9 (which has a preference for small matches) went from 324205506 -> 323999691 bytes.

edit: Found a small improvement for the test, numbers updated.

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)