Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compressing files greater than 2.5 GB trims data #19105

Closed
mukundrv opened this issue Feb 15, 2017 · 4 comments
Closed

Compressing files greater than 2.5 GB trims data #19105

mukundrv opened this issue Feb 15, 2017 · 4 comments

Comments

@mukundrv
Copy link

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.7.5 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="linux"

What did you do?

When I am compressing files greater than 2.5 GB , it successfully creates compressed files but when I again unzip the file and check the data, there is difference in the data. I see the contents missing after a particular line.

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.

A link on play.golang.org is best.
https://play.golang.org/p/P6CnK22q40

What did you expect to see?

I would want the whole content to be gzipped

What did you see instead?

The gzip contains partial content not the full list of lines.

@mvdan
Copy link
Member

mvdan commented Feb 15, 2017

Note that the play link isn't a self-contained test, so the issue isn't reproducible. Perhaps use something like an io.Reader that just gives 3GB of garbage?

@mukundrv
Copy link
Author

The code was given for reference. I don't think it is possible to upload/create files in playground

@mvdan
Copy link
Member

mvdan commented Feb 15, 2017

You don't need to use files. All you need are readers.

@vcabbage
Copy link
Member

I suspect _, err = buffer.Read(rawbytes) is the issue. Read() is not guaranteed to fill rawbytes.

The bytes are taken from at most one Read on the underlying Reader, hence n may be less than len(p).

https://golang.org/pkg/bufio/#Reader.Read

You could use ioutil.ReadAll to get the bytes from the source file, but it would be more efficient and idiomatic to use the io.Reader/io.Writer interfaces to stream the gzip operation. Example: https://play.golang.org/p/u6eAphPRrk

I'm going to close this issue as it does not appear to be a bug, but please feel free to reopen if I've assessed this incorrectly. For any questions on how to use io.Reader/io.Writer, please refer to https://github.com/golang/go/wiki/Questions, where you'll find a number of venues to ask questions.

@golang golang locked and limited conversation to collaborators Feb 15, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants