Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Conduct a community survey on atomic writes #21912

Closed
kevinburke opened this issue Sep 17, 2017 · 5 comments
Closed

proposal: Conduct a community survey on atomic writes #21912

kevinburke opened this issue Sep 17, 2017 · 5 comments

Comments

@kevinburke
Copy link
Contributor

kevinburke commented Sep 17, 2017

In issue #17869 @rgriesemer proposed the idea of adding a primitive to ioutil that atomically writes files. The problem is basically that using the Write() primitive to write a file may destroy an existing file before reporting an error, or that the file may be available to other processes in a partially written state.

The issue was closed because we couldn't reach consensus on the following questions:

  • what permissions should the new file have?
  • where should the temporary file be written (same partition, etc)
  • should we preserve owner and group information? if so, how?
  • should the write call Sync() which incurs a performance cost?

It would be good to do a survey of the community to see how this is currently implemented (or not). Specifically it would be good to answer the following questions:

  • are there many libraries that should be using atomic file writes, that are not? In this case maybe any function in the standard library, even with inappropriate tradeoffs, would be better than nothing.

  • are libraries using atomic writes, but largely in the same way? In this case maybe the standard library function can adopt the primitives used by the community.

  • are people attempting to use atomic file writes, but doing so in a buggy way that exposes them to inconsistencies in the underlying transaction that they're unaware of? For example, the proposal for math.Round revealed that many libraries had errors in their Round implementations. math: add Round #20100

Ideally a community survey could provide some guidance here. Some places to start looking for community "safe write" implementations:

Things to look for:

  • are the implementations using atomic writes at all?
  • do the implementations call Sync()?
  • do the implementations attempt to preserve user/group ownership on write?
  • do the implementations attempt to preserve rwx permissions on write?
  • how do the implementations handle failure in 1) writing the file 2) copying the file?

Maybe a survey would provide some guidance here.

@gopherbot gopherbot added this to the Proposal milestone Sep 17, 2017
@slrz
Copy link

slrz commented Sep 17, 2017

Is there any precedent for this kind of functionality being included into other language's standard libraries or other popular libraries (e.g. Boost for C++)?

@cznic
Copy link
Contributor

cznic commented Sep 17, 2017

POSIX file writes are already atomic, aren't they?

@ianlancetaylor
Copy link
Member

@cznic I assume the issue here is a file that is created with more than one Write call.

@cznic
Copy link
Contributor

cznic commented Sep 18, 2017

Then I suggest to include something like "atomically replace whole file content" in the title. But then we're maybe in the field of journaling and out of scope of the stdlib, I guess.

@rsc
Copy link
Contributor

rsc commented Sep 18, 2017

I think it makes sense to look into and summarize what existing code does. To the extent that you need to ask the community for pointers to existing code, that makes a lot of sense. If you can find the code yourself, that's also fine. You could start with the go-corpus file set or dig more broadly. This doesn't need to be a proposal, though. Feel free to do this if you're interested.

@rsc rsc closed this as completed Sep 18, 2017
@golang golang locked and limited conversation to collaborators Oct 13, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

6 participants