I think this is equivalent to:

package main

import (
	"log"
	"os"
)

func main() {
	f, err := os.Open("in")
	if err != nil {
		log.Fatalln(err)
	}
	out, err := os.OpenFile("out", os.O_APPEND|os.O_CREATE, 0o644)
	if err != nil {
		log.Fatalln(err)
	}
	_, err = out.ReadFrom(f)
	if err != nil {
		log.Fatalln(err)
	}
}

No, I do not think that this is the same.

• I assume, the flags os.O_* are passed directly to the open(2) syscall. There must be one flag for the requested mode: O_RDONLY, O_WRONLY, O_RDWR, at least according to the manpage of open(2) I find on a recent Debian GNU/Linux.

• W/o passing os.O_WR*, I get write: bad filedescriptor, which is ok.

• W/ passing os.O_WRONLY the append mode works.

Even in case this would reproduce the case I'm describing, I wouldn't compare it. In you example the author of the program can control the flags. In my example I can't. I'm just writing to os.Stdout and I can't control if the caller redirects my output using >, >>, or |.

(attn @ianlancetaylor; CC @tklauser @panjf2000)

2023/05/14 10:32:34 0 write /dev/stdout: copy_file_range: bad file descriptor

I think this is because the copy_file_range doesn't support destinations opened with O_APPEND, cf. https://cs.opensource.google/go/go/+/refs/tags/go1.20.4:src/os/readfrom_linux.go;l=15-19 and https://man7.org/linux/man-pages/man2/copy_file_range.2.html#ERRORS.

That check for f.appendMode in the code linked above should cause the fallback path to be taken instead of copy_file_range. However, we don't seem to set File.appendMode in os.NewFile when creating os.Stdout.

This issues seems to be bound to the type of the input. E.g. running the above examples as

cat /etc/hosts | go run . >>/tmp/file-test

I'm not sure why it doesn't fail in that case. I'd still expect O_APPEND to be set on os.Stdout. However, os.Stdin is a pipe in that case, so maybe we're not hitting the path using copy_file_range?

Change https://go.dev/cl/494915 mentions this issue: os: set File.appendMode in newFile if file is opened with O_APPEND

Change https://go.dev/cl/494916 mentions this issue: internal/poll, internal/syscall/unix, net: move and export fcntl syscall wrapper

This issues seems to be bound to the type of the input. E.g. running the above examples as

cat /etc/hosts | go run . >>/tmp/file-test

I'm not sure why it doesn't fail in that case. I'd still expect O_APPEND to be set on os.Stdout. However, os.Stdin is a pipe in that case, so maybe we're not hitting the path using copy_file_range?

Wild guessing w/o reading the (Go) sources: Go tries to be smart if both input and output seem to be regular files. (e.g. >>/dev/null doesn't produce the above error, not sure, if we're not on the copy_file_range path then, or of the OS is forgiving here.)

So, IMHO Go should get even smarter (as suggested by @tklauser) by checking the mode the output is opened as, or less smarter by just avoiding the copy_file_range when operating file handles it doesn't open on its own (0, 1, 2, and maybe even more, if inheritated from a calling process.)

This bug broke my simple program that basically just does "io.Copy(os.Stdout, someData())" whenever that program was used to append to a file: "program >> file.txt" ...

The err you get is: "write /dev/stdout: copy_file_range: bad file descriptor"

was any regression test added to prevent regressions of this in the future?

A basic e2e test like "go run . </etc/hosts >>/tmp/file-test" should definitely be part of the ci build test suite...

We believe that this bug is fixed in the upcoming 1.21 release.

If you want to send in a test, that would be welcome. It could be in the os package, or in cmd/go/testdata/script.

Yes I downloaded 1.21rc3 and appending stdout works there. I will see if I find time to contribute.

Change https://go.dev/cl/517755 mentions this issue: os: test that copying to append-only files doesn't fail on Linux

Change https://go.dev/cl/518255 mentions this issue: os: define TestIssue60181 only on Unix platforms