Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bufio: Read reads at most once #42332

Closed
JordanStoker opened this issue Nov 2, 2020 · 9 comments
Closed

bufio: Read reads at most once #42332

JordanStoker opened this issue Nov 2, 2020 · 9 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@JordanStoker
Copy link

JordanStoker commented Nov 2, 2020

What version of Go are you using (go version)?

go version go1.15.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Ubuntu 18.04 amd64
$ go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jordan/.cache/go-build"
GOENV="/home/jordan/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/jordan/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jordan/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/snap/go/6633"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/snap/go/6633/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build344922599=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Called bufio.Read multiple times until the next read required more bytes than was available i.e. len(buf) > bufio.Available(). When this happened, bufio copied as much as was available in the current buffer but did not attempt to read the remaining bytes requested.

https://play.golang.org/p/M953MGK136s

What did you expect to see?

I expected bufio.Read to recursively perform an io.Read until the requested buffer length was fully read.
Although the documentation does suggest this behavior it is inconsistent with its C-equivalent and the rest of this package - e.g. Discard, write (would write at most twice, w)

What did you see instead?

An incomplete read with no attempt to read the remaining buffer

@Theodoree
Copy link

Theodoree commented Nov 2, 2020

func NewReaderSize(rd io.Reader, size int) *Reader { b, ok := rd.(*Reader) if ok && len(b.buf) >= size { return b } if size < minReadBufferSize { size = minReadBufferSize } r := new(Reader) r.reset(make([]byte, size), rd) return r }
the Reader is struct . not is interface.

@mattn
Copy link
Member

mattn commented Nov 2, 2020

You should make block with two triple-backquote ```.

```
func NewReaderSize(rd io.Reader, size int) *Reader {
    b, ok := rd.(*Reader)
    if ok && len(b.buf) >= size {
        return b
    }
    if size < minReadBufferSize {
        size = minReadBufferSize
    }
    r := new(Reader)
    r.reset(make([]byte, size), rd)
    return r
}
```

@JordanStoker
Copy link
Author

JordanStoker commented Nov 2, 2020

@davecheney If I'm using a bufio.Reader to read a large data file I will get some truncated reads.

To work around this, I'd need to check for the bufio.Read output for truncated read and then call the function again... In C I wouldn't have to do this myself, the Read function would do it for me. It would be a straight forward addition to the package

@davecheney
Copy link
Contributor

@JordanStoker i'm sorry, my initial comment was incorrect. I deleted it, I'm sorry.

@davecheney
Copy link
Contributor

To work around this, I'd need to check for the bufio.Read output for truncated read and then call the function again... In C I wouldn't have to do this myself, the Read function would do it for me. It would be a straight forward addition to the package

The short version is, io.Reader doesn't guarentee to read anything more than 1 character, in fact, it can read 0 characters without error.

@Theodoree
Copy link

@mattn Thanks for reminding

@toothrot toothrot changed the title bufio.Read bufio: Read reads at most once Nov 2, 2020
@toothrot
Copy link
Contributor

toothrot commented Nov 2, 2020

I am not sure how this could be changed with the compatibility promise.

I might be missing something, could you clarify how the documented suggestion of using io.ReadFull doesn't help in your scenario?

/cc @griesemer @bradfitz @ianlancetaylor

@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 2, 2020
@toothrot toothrot added this to the Backlog milestone Nov 2, 2020
@JordanStoker
Copy link
Author

@toothrot In my scenario, I am unpacking small data structures from a large file.
Loading the whole file into memory isn't feasible and reading just the bytes I need for each small structure would be very slow - too many System calls. The compromise is to read data in manageable chunks, which is where the bufio package is helpful.
However, if my next read is larger than the remaining bytes available in the bufio.Reader buffer then my data structure gets truncated.

Here is a workaround I've used where I would recursively call the bufio.Reader until my data has been fully read. Recursion is overkill in my scenario (my structures are much much smaller than the bufio buffer), a single bufio.Reader.Read would have sufficed.

func readBuf(r *bufio.Reader, p []byte) (nn int, err error) {
	nn, err = r.Read(p)
	if err != nil {
		return
	}
	if nn < len(p) {
		var n int
		n, err = readBuf(r, p[nn:])
		nn += n
	}
	return
}

For the opposite scenario where I want to pack data in a file, the bufio.Writer.Write (https://golang.org/src/bufio/bufio.go) would iteratively write the data if the data is larger than the buffer available. I would have expected the same behaviour from bufio.Reader.

@davecheney
Copy link
Contributor

@JordanStoker it looks like you can replace your readBuf function with https://golang.org/pkg/io/#ReadFull as readBuf reads up to len(p) bytes or until an error occurs.

@golang golang locked and limited conversation to collaborators Nov 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

6 participants