New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
compress/gzip: expose sub-file information #6486
Labels
Milestone
Comments
Sorry, that is obviously the diff quoted at https://groups.google.com/d/msg/golang-dev/nA3QRGIvrOE/cXv7ygTsU8IJ. What I'm interested in knowing is if the change that comes is likely to be in line with what was tentatively proposed there. I am asking because I have been holding off work on my package waiting for this issue, so now that it looks like if will possibly be another three to six months (depending on how the release cycle is dealt with this round), I don't think I can wait for the standard library any more. |
Thanks, Russ. I should clarify. I have a fork that uses my approach to handling the member header to deal with blocked decoding, but maintaining this while bringing in changes from compress/gzip (and keeping public-facing types and fields interoperable with that package) while not difficult it painful. I was hoping to retire that fork when the fix for this issue and the (otherwise unrelated) CL13512052 goes in, before moving on to deal with the long term implementation of the BGZF reader which will be doing partially parallel blocked read ahead with block caching. This requires careful handling of gzip member header fields that I'm only just able to keep my head around in a stable world, changing API is likely to blow away my capacity to handle that. |
I have come back to this code since I have found some time with clear thought. I can now see that I can achieve exactly what I want to do with the API as is stands at the moment. Apologies for confusion. To explain, the current gzip.Reader Read() method returns with a nil error without filling the parameter slice when the call to z.decompressor.Read(p) reads across a member boundary (because err != io.EOF since the decompressor return nil here). In my case I intend to already know the start of the next member, so a gzip.Reader Reset() on the underlying Reader Seek'd to the appropriate location will give me the next Header. The upshot is that a test of n < len(p) && err == nil will tell you if you have reached a boundary. After a brief reading of the WARC format mentioned by Daniel Krech in the original thread, I think they should also be able to use this approach. |
CL https://golang.org/cl/159120044 mentions this issue. |
This issue was closed by revision 70f2f1b. Status changed to Fixed. |
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jun 25, 2018
Allows parsing some file formats that assign special meaning to which stream data is found in. Will do the same for compress/bzip2 once this is reviewed and submitted. Fixes golang#6486. LGTM=nigeltao R=nigeltao, dan.kortschak CC=adg, bradfitz, golang-codereviews, r https://golang.org/cl/159120044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jun 26, 2018
Allows parsing some file formats that assign special meaning to which stream data is found in. Will do the same for compress/bzip2 once this is reviewed and submitted. Fixes golang#6486. LGTM=nigeltao R=nigeltao, dan.kortschak CC=adg, bradfitz, golang-codereviews, r https://golang.org/cl/159120044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jul 9, 2018
Allows parsing some file formats that assign special meaning to which stream data is found in. Will do the same for compress/bzip2 once this is reviewed and submitted. Fixes golang#6486. LGTM=nigeltao R=nigeltao, dan.kortschak CC=adg, bradfitz, golang-codereviews, r https://golang.org/cl/159120044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jul 20, 2018
Allows parsing some file formats that assign special meaning to which stream data is found in. Will do the same for compress/bzip2 once this is reviewed and submitted. Fixes golang#6486. LGTM=nigeltao R=nigeltao, dan.kortschak CC=adg, bradfitz, golang-codereviews, r https://golang.org/cl/159120044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jul 30, 2018
Allows parsing some file formats that assign special meaning to which stream data is found in. Will do the same for compress/bzip2 once this is reviewed and submitted. Fixes golang#6486. LGTM=nigeltao R=nigeltao, dan.kortschak CC=adg, bradfitz, golang-codereviews, r https://golang.org/cl/159120044
This issue was closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The text was updated successfully, but these errors were encountered: