Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

io: clarify Writer bytes written semantics #56533

Open
kevinburke opened this issue Nov 2, 2022 · 5 comments
Open

io: clarify Writer bytes written semantics #56533

kevinburke opened this issue Nov 2, 2022 · 5 comments
Labels
Documentation NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@kevinburke
Copy link
Contributor

The documentation for io.Writer states:

Writer is the interface that wraps the basic Write method.

Write writes len(p) bytes from p to the underlying data stream. It returns the number of bytes written from p (0 <= n <= len(p)) and any error encountered that caused the write to stop early. Write must return a non-nil error if it returns n < len(p). Write must not modify the slice data, even temporarily.

And for example, *os.File.Write() says this:

Write writes len(b) bytes from b to the File. It returns the number of bytes written and an error, if any. Write returns a non-nil error when n != len(b).

With some write API's, to sockets or files, you may get an error and not know whether the write actually succeeded or not (see https://danluu.com/file-consistency/ for examples).

If, say, 16 bytes are written and there is an error writing byte 17. The interface description isn't clear about what should happen:

  • Write should always report (16, error)
  • Write should try to determine whether the error indicates no data was written (and return 16), otherwise return 17
  • Write should always report (17, error)
  • Write can return 16 or 17 as it likes.

Under the hood, *os.File.Write for example eventually invokes the write syscall, which also does not give any clarity about what happens, at least on BSD.

Upon successful completion the number of bytes which were written is returned. Otherwise, a -1 is returned and the global variable errno is set to indicate the error.

My request is to clarify what should happen, and then document it for consistency.

@ianlancetaylor
Copy link
Contributor

If the system call doesn't tell us what happened, then the Write method can't tell the caller what happened. So I'm not sure what we should document. "It would be nice if we reported this correctly but we can't promise?"

I think the intent is clear: Write should report the number of bytes that were actually written. But if we don't know, we don't know.

@kevinburke
Copy link
Contributor Author

But if we don't know, we don't know.

Fair; if we don't know, then what do we report? Should the count returned by Write include all of the bytes that were attempted, or none? That's what I'm trying to document, I guess.

@ianlancetaylor
Copy link
Contributor

I see. I think none. While we don't know whether that is accurate, it seems more likely than all or some, I think.

@seankhliao seankhliao added Documentation NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 2, 2022
@extemporalgenome
Copy link
Contributor

Much of Go's writer implementations or wrappers expect reliable byte stream: errors are often assumed to imply that the writer is unusable past that point.

The returned count from a Read call is particularly meaningful since Read can return early without error. In contrast, a Write call is expected to block until the full buffer has been written.

For writer-as-stream cases, it's not clear how valuable the returned n value would be. For seekable writers, it may be useful for tracking file offset, though WriterAt is probably more useful in those cases.

Since we should consistently return some value for n, due to the previously discussed ambiguity, perhaps we should just return the number of bytes affirmatively reported as being successfully written (which may be less than the number of bytes actually received, such as in the case of a broken network connection).

In the original scenario listed, this may mean that we report 0 bytes written, since the underlying transport may be unable to distinguish the number of bytes received on the other side and because we're sending all 17 bytes at once (we certainly wouldn't be writing 1 byte at a time just to have byte-level failure precision). If, internally, we sent a batch of 16 bytes (e.g. due to framing), which succeeded, and then sent 1 byte, which failed, we would certainly return 16, since that number was affirmatively successful.

@bcmills
Copy link
Contributor

bcmills commented Nov 3, 2022

I think it would be more useful to document the semantics of specific errors than to try document the semantics of n independent of the error value.

If there are specific errors that always indicate an exact count of bytes written — or specific errors that always indicate an inexact count of bytes written — then it seems useful for callers who observe those specific errors to know those implications, but given the ambiguity in the io.Writer interface — and the fact that existing implementations may have interpreted it either way — in the absence of a specific error value that says otherwise (or documentation for a specific implementation of the Write method) I don't think the caller can safely assume anything about the number of bytes actually written.

@seankhliao seankhliao added this to the Unplanned milestone Nov 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants