Skip to content

proposal: io: io.WriterToAt and io.ReaderFromAt #52383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Jorropo opened this issue Apr 16, 2022 · 6 comments
Closed

proposal: io: io.WriterToAt and io.ReaderFromAt #52383

Jorropo opened this issue Apr 16, 2022 · 6 comments
Labels
Milestone

Comments

@Jorropo
Copy link
Member

Jorropo commented Apr 16, 2022

Proposal

WriterToAt

// Start gives you the current head point, length is how much data is remaining.
func (*SectionWriter) Unwrap() (r WriterAt, start int64, length int64)

type WriterToAt interface {
  // WriteToAt SHOULD be capable to unpack *SectionWriter objects to accelerate copies.
  // If a copy from an unwrapped *SectionWriter is performed, *SectionWriter also need to be seeked to reflect that.
  // Off is the offset in the reader. If you want pass an offset in the Writer, wrap it in an *SectionWriter first.
  // If the length missmatch between the reader and the writer copy until the smallest one and return EOF.
  // If the copy could not be performed more efficiently than with a simple read write copy loop, then return with errors.ErrUnsupported
  // If the copy was succesfull err == nil and n == length.
  // If length == -1 that means copy until EOF.
  WriteToAt(w Writer, off int64, length int64) (n int64, err error)
}

ReaderToAt (exactly the same but reader and writer are swapped)

// Start gives you the current head point, length is how much data is remaining.
func (*SectionReader) Unwrap() (r ReaderAt, start int64, length int64)

type ReaderFromAt interface {
  // ReadFromAt SHOULD be capable to unpack *SectionReader objects to accelerate copies
  // If a copy from an unwrapped *SectionReader is performed, *SectionReader also need to be seeked to reflect that.
  // Off is the offset in the reader. If you want pass an offset in the Reader, wrap it in an *SectionReader first.
  // If the length missmatch between the writer and the reader copy until the smallest one and return EOF.
  // If the copy was succesfull err == nil and n == length.
  // If the copy could not be performed more efficiently than with a simple read write copy loop, then return with errors.ErrUnsupported
  // If length == -1 that means copy until EOF.
  ReadFromAt(w Reader, off int64, length int64) (n int64, err error)
}

Motivations

Clean (not having to deal with FDs and CopyFileRange) concurrent zerocopy.
On linux, at least, concurrent reads are writes are perfectly safe and sane as long as no writes overlap and no reads overlap with a write.

Easly composable offset based zero copy.
For example a database that use a single big file could return *SectionReader to objects pointing to objects on the disks. And even if multiple step later (after being wrapped io.MultiReader call for example), zerocopy would still just work.

Generalisable, other types like bytes.Reader could be optimised that way too.

Current WriterTo and ReaderFrom allows similar operations if you combine them with seeking and LimitedReader but that is not concurrent safe.

The reason to use io.Section* types as wrappers that way is because this avoid an big (8) matrix of required interfaces:

{ReaderFrom,WriterTo}{At,}({Writer,Reader}{At,})

With this only 4 interfaces are required. 2 of them are already in the std already.
We replace the type matrix with type assertion in the different implementations.

Related issues:

@ianlancetaylor
Copy link
Member

Can we get the same effect by implementing io.SectionReader.WriteTo and io.SectionWriter.ReadFrom?

@Jorropo

This comment has been minimized.

@Jorropo

This comment has been minimized.

@rsc rsc moved this to Incoming in Proposals Aug 10, 2022
@rsc rsc added this to Proposals Aug 10, 2022
@CAFxX
Copy link
Contributor

CAFxX commented Jul 21, 2023

Recently ran into this while attempting to have multiple readers read (via io.Copy) from the same os.File in parallel. Wrapping the os.File in a SectionReader should do the trick, but that currently implies giving up on the WriterTo fast path, and in general on any hope of zero-copy send.

@Jorropo
Copy link
Member Author

Jorropo commented Oct 30, 2024

Found a more practical way to do this.

@Jorropo Jorropo closed this as not planned Won't fix, can't repro, duplicate, stale Oct 30, 2024
@nightlyone
Copy link
Contributor

@Jorropo could you share the more practical approach here please, so people tracking or finding this issue can learn the alternative solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants