New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
io/ioutil: WriteFile does not write content to file #14491
Comments
Please show us a complete test case that reproduces the problem. It is never necessary to call Sync on a stable system. On GNU/Linux calling Sync just calls fsync. |
@TheHippo to complete your example where atleast the paths match I've written http://play.golang.org/p/v9Dd3dJYMH -- content shown below. package main
import (
"bytes"
"fmt"
"io/ioutil"
"log"
"os"
)
func reproTrial(path string, content []byte) error {
err := ioutil.WriteFile(path, content, 0644)
if err != nil {
return err
}
defer func() {
_ = os.RemoveAll(path)
}()
retrieved, err := ioutil.ReadFile(path)
if err != nil {
return err
}
if !bytes.Equal(retrieved, content) {
return fmt.Errorf("file %q content doesn't match, read %s expected %s\n", path, retrieved, content)
}
return nil
}
func main() {
outfilePath := "temp"
content := []byte("foo")
if err := reproTrial(outfilePath, content); err != nil {
log.Fatal(err)
}
} |
I can't reproduce either (Ubuntu 12.04 ssd, Amazon Linux EBS and ephemeral).
|
After some digging a few more facts about when/how this happens:
Writing the file "manually" and calling |
@TheHippo What triggers the signal sent to the C process? Is the Go process sending it after the If you're able to provide a minimal but complete code sample that reproduces the issue (even if only on your loaded system), that would be helpful. |
@TheHippo what FS? |
There was nothing on the TV so I tried to reproduce it with an external process reading the file but couldn't. Even with the high IO load.
Reader:
|
I ran into the same issue on 3.19.0-32-generic x86_64 GNU/Linux. Linux Mint Rosa 17.3, based on Ubuntu 14.04.3 LTS. Filesystem ext4 on LVM on LUKS encrypted drive. go version go1.6 linux/amd64 My code parses logfiles and for testing writes each hit to a file in the file system: During a test run I got this:
Code responsible for the panic:
Immediately after writing the file, I stat it to get the size. Cop out I know, but it demonstrates the point TheHippo is making. Looking at it a few seconds later:
If I'm allowed an opinion, I think a convenience and utility method WriteFile should guarantee that the file exists after it has completed and thus should call Sync. If you want performance you code it "manually" and take control of the Syncronization or not. |
your code didn't check the error returned from ioutil.WriteFile.
|
I'm scrubbing bugs and am going to close this until there's something for us to fix. Like @minux said, please check your errors. They usually contain the answers. |
As stated in the comment, the file does get created, I know it doesn't check the error code. All well and good but that is cheap deflection: You don't address the statement made that a convenience and utility method like WriteFile should guarantee that the file exists when it returns. It simply doesn't. I'm not asking for you or anybody to agree with me, but at least take a stance for or against. |
I think that is the root of the confusion. Go is from a different style of languages compared to, say, Java, with which perhaps you're more familiar. In Go, a function may not do what it says if it returns an error. Here are 821 functions and methods from the Go standard library which are no different from Always check your errors. |
With that statement I agree and admittedly, the code is not production quality. If this were a code review the hammering on checking my errors would be warranted. But that is in this particular case not the point. There is only one spot in the code in which it creates the file which the panic claims is missing. Interrupting the test run and checking the whether the file really doesn't exist, I find the file is present and it is complete and contains what it should be containing. Still the panic just claimed it didn't exist. Although I cannot claim with absolute certainty that no error occurred I think I am reasonably sure that no error was returned, since the file is created and complete and correct. Unless there is a possibility that the function returns an error and does what it is supposed to do anyway, just a little time after returning (? Don't know how, but I would be interested in your explanation ?). Leaves me with a timing issue during the interaction between the go run-time and the operating system. WriteFile thinks it has created the file (or not depending on your explanation) and returns. The os appears not to immediately write the file, might be busy with some other task (it is busy creating about 12000 files in about 1 minute). os.Stat panics b/c the file does not exists, I check after hitting CTRL-C and it turns out the file does exists and contains exactly what the expectation was. I think this confirms exactly what TheHippo claimed in his initial opening of this issue. I will code around this and avoid using ioutil.WriteFile, since it is not reliable. Wilbert |
@dotwilbert, WriteFile is not special at all. You should not avoid it. If WriteFile succeeds without error, Stat should succeed too, immediately after it. If not, it's a problem with the filesystem or kernel or Go runtime. It's not something you can work around by changing which functions you use. If anybody has a reliable repro of this, please file a new bug with details of the operating system, version, filesystem type, and Go code checking all its errors. |
Sounds fair. I will run multiple test runs. I'll check the errors and let you know when if it happens again. |
After carefully reviewing the testing procedure, I discovered an error on the command line. In the particular test above I processed more than the intended one day in a bash loop. This gives an alternate explanation for the existence of the file that the panic was about. I retract the remark about WriteFile's unreliability and commend you for standing by it. I'll continue testing with error checking enabled. Thanks, Wilbert |
Using Go 1.5.3 on Linux/amd64:
Writing a file like:
Reading the file content of
temp
immediately after writing it sometimes indicates the content hasn't been written to the file. Looking at the source code ofWriteFile
it seems like a call toSync()
is missing.P.S.: Tested with larger content.
The text was updated successfully, but these errors were encountered: