Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bytes: it is impossible to Seek() in slice with len more than 2^32 bytes #7654

Closed
gopherbot opened this issue Mar 27, 2014 · 7 comments
Closed
Milestone

Comments

@gopherbot
Copy link

by marko@kevac.org:

This bug is in all go versions (including tip).

It is impossible to Seek() in bytes.Reader to offset higher than 2^32 because of the
check inside Seek() implementation.

This check is not needed IMHO. Latest Go versions allow slices with length larger than
2^32.

Patch would be very simple. Just remove check from Seek() implementation.
@cznic
Copy link
Contributor

cznic commented Mar 27, 2014

Comment 1:

The specs allow slices with indices/len/cap of only int sizes. Note that 'int' is
architecture specific. Allowing bytes.Reader to accommodate buffers greater than or
equal to 1<<31 (not 2^32, btw) will enable writing programs which would run on 8
byte int archs but would not run on architectures having only 4 byte ints. Also, for
example Reader's Len would have no way how to return a correct value in some cases on
such archs. IOW, this cannot be changed, AFAICT.
#WAI

@gopherbot
Copy link
Author

Comment 2 by marko@kevac.org:

Right now bytes does not work for 64bit archs.
Would it be correct if we change the check to use unsafe.Sizeof instead of using
1<<31?

@bradfitz
Copy link
Contributor

Comment 3:

Yeah, this is surprising. strings.Reader, too.
I think we could change this and make it bounded by the size of int on the local
machine, not 1<<31.
No point giving you 64-bit ints and slice/string lengths and be unable to use it all
here.  There are already Go programs that only run on 64-bit machines (programs needing
lots of memory), so I'm not concerned about that.

Labels changed: added release-go1.3maybe.

Status changed to Accepted.

@cznic
Copy link
Contributor

cznic commented Mar 27, 2014

Comment 4:

Seek(1<<33) would then compile only on 8 byte int archs. I know that the same
problem exists for _ = slice[1<<33], but I think such code should discouraged by
the stdlib. Also a 2GB buffer is big enough for anything I can imagine. The purpose of
buffer is not to accommodate _all_ data but to make processing the data faster when
interleaved I/O and processing is the bottleneck. The improvement often happens with
buffers as small as only few kB. Then the law of diminishing results inevitably steps in
and huge buffers can even make things slower wrt to latency of some initial result data
being produced.

@gopherbot
Copy link
Author

Comment 5 by marko@kevac.org:

I have stumbled upon this issue when trying to use bytes.Reader on []byte returned from
syscall.Mmap. Are you saying that having files larger than 2GB is impractical? :-)

@bradfitz
Copy link
Contributor

Comment 6:

Sent https://golang.org/cl/81530043

Status changed to Started.

@bradfitz
Copy link
Contributor

Comment 7:

This issue was closed by revision f074565.

Status changed to Fixed.

@rsc rsc added this to the Go1.3 milestone Apr 14, 2015
@golang golang locked and limited conversation to collaborators Jun 25, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants