Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compress/gzip: document that Flush writes additional data, even if there is no more compressed data to write #16068

Closed
pavelnikolov opened this issue Jun 15, 2016 · 4 comments
Closed

compress/gzip: document that Flush writes additional data, even if there is no more compressed data to write #16068

pavelnikolov opened this issue Jun 15, 2016 · 4 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.8

Comments

@pavelnikolov
Copy link

I find it confusing that the gzip.Writer.Flush() method doesn't actually flush the compressed data to the underlying buffer.
I'm running: go version go1.6.1 darwin/amd64

Here is the code: https://play.golang.org/p/zXIh2ZL5Cs

I was expecting the number of written bytes after flushing and after closing the writer to be the same but that is not the case.

Also calling Flush() multiple times results in more data being written to the underlying buffer than it was written to the gzip writer – https://play.golang.org/p/ochlIkK6KS – is this expected behavior or is it a bug?
@adg
Copy link
Contributor

adg commented Jun 15, 2016

It is expected that Close should write data in addition to the final Flush, as it must conclude the gzip stream with a checksum of its contents.

As for Flush writing additional data, the docs say:

In the terminology of the zlib library, Flush is equivalent to Z_SYNC_FLUSH.

A google search for Z_SYNC_FLUSH yields this page which contains a description of Z_SYNC_FLUSH and it does indeed involve writing extra data.

I don't think we want to change this behavior, as people use gzip streams in weird and wonderful ways. Maybe we should document it more?

@adg adg added this to the Go1.8 milestone Jun 15, 2016
@adg adg changed the title compress/gzip Flush doesn't actually flush the writer compress/gzip: document that Flush writes additional data, even if there is no more compressed data to write Jun 15, 2016
@pavelnikolov
Copy link
Author

Thank you for the quick response! Yeah, better documentation would be very useful as I would expect Flush() to behave differently and probably I'm not alone.

@adg adg added the NeedsFix The path to resolution is known, but the work has not been done. label Jun 15, 2016
@dsnet
Copy link
Member

dsnet commented Jun 15, 2016 •
edited

What @adg said is correct. The behavior of gzip flushing is necessary for the correct operation of some network protocols. Not only does Flush ensure all buffered data has been flushed to the wire, it also writes a special synchronization message (really just an empty raw block) for the recipient to know that such a flushing event had occurred. There's an entire unit test in compress/flate just to lock this behavior in.

Just to confirm though, Flush does indeed ensure all the buffered data has hit the underlying io.Writer.

I can document this.

@dsnet dsnet self-assigned this Jun 15, 2016
@gopherbot
Copy link

CL https://golang.org/cl/28477 mentions this issue.

@golang golang locked and limited conversation to collaborators Sep 4, 2017
@rsc rsc unassigned dsnet Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.8
Development

No branches or pull requests
4 participants
@pavelnikolov @dsnet @adg @gopherbot