Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

io: ReadAll doesn't respect io.ErrShortBuffer #48432

Closed
utrack opened this issue Sep 17, 2021 · 3 comments
Closed

io: ReadAll doesn't respect io.ErrShortBuffer #48432

utrack opened this issue Sep 17, 2021 · 3 comments

Comments

@utrack
Copy link

utrack commented Sep 17, 2021

What version of Go are you using (go version)?

$ go version
go version go1.17.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

(not relevant)

What did you do?

I create an io.Reader that sometimes returns io.ErrShortBuffer. Then, I try to slurp data from it via io.ReadAll().

What did you expect to see?

io.ReadAll() manages its buffer properly and retries r.Read() with a bigger buffer.

What did you see instead?

io.ReadAll() returns io.ErrShortBuffer on a first ErrShortBuffer encounter.

Notes

I believe that it should handle the ErrShortBuffer gracefully by default.

There's a memory management problem: faulty Reader might spam ErrShortBuffer and the buffer would grow exponentially; however, ReadAll doesn't care about that by default since the source itself is unbounded and data can outgrow available memory pretty easily.

Edit: oh, I've noticed that many reader implementations incl. bufio, bytes etc don't support ErrShortBuffer as well. Is there any canonical way to signal upstream that a buffer must be grown?

@ericlagergren
Copy link
Contributor

ericlagergren commented Sep 17, 2021

ErrShortBuffer is only used by ReadAtLeast.

Also, you shouldn’t need ErrShortBuffer with Reader. It’s legal to fail if len(p) is less than some desired buffer size, but non-standard as Readers conventionally return partial data instead of no data.

@utrack
Copy link
Author

utrack commented Sep 17, 2021

The problem with partial reads with small buffer is that they aren't actually possible with the current io.ReadAll() implementation.

https://play.golang.org/p/9e5bcYIb_1n

For a first read, partialReader has a buffer of 512 bytes. If it returns 256 then on the next read io.ReadAll() gives a buffer of only 256 bytes. If it wasn't enough on a first read then it won't get any bigger on the second one.

There's just no way for a downstream Reader to signal to upstream that this buffer isn't big enough for the read.

@ianlancetaylor
Copy link
Contributor

There's just no way for a downstream Reader to signal to upstream that this buffer isn't big enough for the read.

The io.Reader API does not provide any mechanism for that. The expectation is that any type that implements io.Reader and can only read entire chunks at a time will do its own buffering. There is no other way to honor the io.Reader interface.

As @ericlagergren says, an implementation of io.Reader should never return io.ErrShortBuffer.

I'm going to close this issue because I don't think there is anything to change here. Please comment if you disagree.

@golang golang locked and limited conversation to collaborators Sep 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants