Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bufio: Writer failures forever even if the temporary error is recoverd #44120

Closed
rickif opened this issue Feb 5, 2021 · 2 comments
Closed

bufio: Writer failures forever even if the temporary error is recoverd #44120

rickif opened this issue Feb 5, 2021 · 2 comments
Labels
FrozenDueToAge

Comments

@rickif
Copy link

rickif commented Feb 5, 2021 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.15.6 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ricky/.cache/go-build"
GOENV="/home/ricky/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ricky/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ricky/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build940457028=/tmp/go-build -gno-record-gcc-switches"

What did you do?

package main

import (
	"bufio"
	"log"
	"os"
	"strings"
	"time"
)

var buf = make([]byte, 1024*1024)

func main() {
	fh, err := os.OpenFile("bufio.file", os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0777)
	if err != nil {
		log.Fatal(err)
	}
	bw := bufio.NewWriterSize(fh, 16*1024)
	var cnt int64
	for {
		n, err := bw.Write(buf)
		if err != nil {
			if strings.Contains(err.Error(), "no space") {
				time.Sleep(time.Second)
				log.Println("no space, sleep 1 second")
				if err := bw.Flush(); err != nil {
					log.Println(err)
				} else {
					log.Println("recovered")
				}
				continue
			}
			log.Println(err)
		}
		cnt += int64(n)

		if cnt%(1024*1024*1024) == 0 {
			log.Printf("%v written\n", cnt)
		}
	}
}

I run program built from codes above on a file system with little size left. Soon, the program logs error write bufio.file: no space left on device. Then I clear some disk files and make room for the program writes, but the program still gets error write bufio.file: no space left on device.

go/src/bufio/bufio.go

Lines 633 to 655 in 8869086

func (b *Writer) Write(p []byte) (nn int, err error) {
for len(p) > b.Available() && b.err == nil {
var n int
if b.Buffered() == 0 {
// Large write, empty buffer.
// Write directly from p to avoid copy.
n, b.err = b.wr.Write(p)
} else {
n = copy(b.buf[b.n:], p)
b.n += n
b.Flush()
}
nn += n
p = p[n:]
}
if b.err != nil {
return nn, b.err
}
n := copy(b.buf[b.n:], p)
b.n += n
nn += n
return nn, nil
}

go/src/bufio/bufio.go

Lines 600 to 621 in 8869086

func (b *Writer) Flush() error {
if b.err != nil {
return b.err
}
if b.n == 0 {
return nil
}
n, err := b.wr.Write(b.buf[0:b.n])
if n < b.n && err == nil {
err = io.ErrShortWrite
}
if err != nil {
if n > 0 && n < b.n {
copy(b.buf[0:b.n-n], b.buf[n:b.n])
}
b.n -= n
b.err = err
return err
}
b.n = 0
return nil
}

I dig into the golang source code and get the reason: the Writer's err is set at the first time when the no space left on device returned. Moreover, there's no method to clear the error unless call Writer.Reset(), which would also clear the data that the Writer buffed.

What did you expect to see?

When the file system capacity is OK, retry to calling Writer.Write()/Flush() and the failure would recover.

What did you see instead?

The failure exists forever even if the file system capacity is OK.
@rickif
Copy link
Author

rickif commented Feb 5, 2021 •
edited

It is common case to handle temporary failure in IO. Is it better to try to do write every time the Writer.Write()/Flush() is called than just to return the stale error simply ?

@rickif rickif changed the title bufio: Writer failures forever even if enviroment's temporary error is recoverd bufio: Writer failures forever even if the temporary error is recoverd Feb 5, 2021
@ianlancetaylor
Copy link
Contributor

The way that the bufio package handles write errors is by design, so that it is not necessary for callers to check the error result on every call to Write. This is documented at https://golang.org/pkg/bufio/#Writer. There is no general way to address this, as the current library doesn't provide a way to distinguish between permanent and temporary errors on Write.

The result of these decisions is that the bufio package is not suitable for your purpose. You should either use your own buffering package instead of bufio, or you should use your own io.Writer that handles temporary errors in the Write method called by bufio.

@golang golang locked and limited conversation to collaborators Feb 5, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ianlancetaylor @gopherbot @rickif