You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The specific motivating case here is when doing decoding from a stream of json. In some cases I need to make two passes to determine the appropriate struct into which to decode based on some content. The json.Decoder offers buffered in an io.Reader. Thus I have access to the in-tact stream by taking:
io.MultiReader(dec.Buffered(), r) // where r is the underlying reader.
The problem with this is that each time this process happens, a new multiReader is created to wrap these two readers. In general, the next call to read will Read the entire Reader from Buffered leaving just the underlying reader. The problem is that the last reader remains wrapped in an underlying *multiReader. So after doing this 100 times, there may be just one actual reader with data but the call to read will have to call through the multiReader.Read method 100 times.
func MultiReader(readers ...Reader) Reader {
r := make([]Reader, 0, len(readers))
for _, reader := range readers {
if mr, ok := reader.(*multiReader); ok {
r = append(r, mr.readers...)
} else {
r = append(r, reader)
}
}
return &multiReader{r}
}
The text was updated successfully, but these errors were encountered:
bradfitz
changed the title
io.MultiReader should flatten multiReaders at construction time
io: MultiReader should flatten multiReaders at construction time
Dec 9, 2015
if len(mr.readers) == 1 {
if r, ok := mr.readers[0].(*multiReader); ok {
mr.readers = r.readers
continue
}
}
n, err = mr.readers[0].Read(p)
if n > 0 || err != EOF {
if err == EOF {
// Don't return EOF yet. There may be more bytes
// in the remaining readers.
err = nil
}
return
}
mr.readers = mr.readers[1:]
}
return 0, EOF
}
This is probably better because it doesn't change the runtime of the function. The previous solution could make a sequence of calls to MultiReader with non-MultiReaders expensive, like when combining many short buffers. It would become O(readers contained) instead of O(readers passed), the latter of which is probably more expected. It also eliminates the possibly bad runtime from the successive calls to nested multiReaders that contain few actual Readers that originally prompted the proposal.
ajwerner
changed the title
io: MultiReader should flatten multiReaders at construction time
io: MultiReader should be more efficient when chained multiReaders contain few actual readers
Dec 15, 2015
The specific motivating case here is when doing decoding from a stream of json. In some cases I need to make two passes to determine the appropriate struct into which to decode based on some content. The json.Decoder offers buffered in an io.Reader. Thus I have access to the in-tact stream by taking:
io.MultiReader(dec.Buffered(), r) // where r is the underlying reader.
The problem with this is that each time this process happens, a new multiReader is created to wrap these two readers. In general, the next call to read will Read the entire Reader from Buffered leaving just the underlying reader. The problem is that the last reader remains wrapped in an underlying *multiReader. So after doing this 100 times, there may be just one actual reader with data but the call to read will have to call through the multiReader.Read method 100 times.
The current constructor looks like this:
I propose that we do something like this:
The text was updated successfully, but these errors were encountered: