New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: io: io.MultiReader should read until the buffer is full or there are no Readers left #57750
Comments
The current code seems OK and implements the documented semantics for |
@ianlancetaylor Thank you for your reply. When using io.Copy or io.CopyBuffer, it is often desirable to fill the buffer as much as possible before writing (except for the last write), the reason being to align memory and disk (probably when using direct-io) for more efficient write performance. When using direct-io, in order to align 4KB, it may be necessary to first read a portion of the file to form a multiple of 4KB with the newly added data. In this scenario, the two parts can form a MultiReader, and io.Copy does not care if the buffer is full when writing. |
sounds like a case where should wrap an |
This is a part of bufio source code. @seankhliao // Read reads data into p.
// It returns the number of bytes read into p.
// The bytes are taken from at most one Read on the underlying Reader,
// hence n may be less than len(p).
// To read exactly len(p) bytes, use io.ReadFull(b, p).
// If the underlying Reader can return a non-zero count with io.EOF,
// then this Read method can do so as well; see the [io.Reader] docs.
func (b *Reader) Read(p []byte) (n int, err error) {
n = len(p)
if n == 0 {
if b.Buffered() > 0 {
return 0, nil
}
return 0, b.readErr()
}
if b.r == b.w {
if b.err != nil {
return 0, b.readErr()
}
if len(p) >= len(b.buf) {
// Large read, empty buffer.
// Read directly into p to avoid copy.
n, b.err = b.rd.Read(p)
if n < 0 {
panic(errNegativeRead)
}
if n > 0 {
b.lastByte = int(p[n-1])
b.lastRuneSize = -1
}
return n, b.readErr()
}
// One read.
// Do not use b.fill, which will loop.
b.r = 0
b.w = 0
n, b.err = b.rd.Read(b.buf)
if n < 0 {
panic(errNegativeRead)
}
if n == 0 {
return 0, b.readErr()
}
b.w += n
}
// copy as much as we can
// Note: if the slice panics here, it is probably because
// the underlying reader returned a bad count. See issue 49795.
n = copy(p, b.buf[b.r:b.w])
b.r += n
b.lastByte = int(b.buf[b.r-1])
b.lastRuneSize = -1
return n, nil
} As I understand the code, bufio.Reader only reads from the underlying reader once at a time, which is not enough to ensure that the buffer is filled. I think that io.MultiReader should make multi files as one file to the caller. If a file can be read again to fill the buffer, then it should also do it when it is divided into multiple parts and combined by the io.MultiReader. |
MultiReader can be used even for TCP streams and many more. If you call Read on anything it must block until it returns at least a single byte (or error). |
@polomsky maybe you are right |
This proposal has been added to the active column of the proposals project |
This proposal has been declined as infeasible. |
It's an example
The text was updated successfully, but these errors were encountered: