Are you sure that the error message is "function not implement" ? I don't see where that could come from. Could it be "function not implemented"?

What Linux kernel version are you running?

Yes, it's "implemented" might have cut the 'ed' on accident.

┌─[root]
└─▪ go run main.go
runtime: copying /usr/lib/go-1.20/src/runtime/asm_amd64.h to /tmp/go-build682241010/b008/asm_GOARCH.h: write /tmp/go-build682241010/b008/asm_GOARCH.h: copy_file_range: function not implemented

uname -sr: Linux 5.4.0

same issue. reverting to go 1.19.4 "fixes it"

I'm able to reproduce this, it seems to be a regression introduced in Go 1.20.0. To expand on @remover's comment: the exact message in our case is:

runtime: copying /usr/local/go/src/runtime/tls_arm64.h to /tmp/go-build4200235085/b002/tls_GOARCH.h: write /tmp/go-build4200235085/b002/tls_GOARCH.h: copy_file_range: function not implemented

In our case, we see the error when running go run in an arm64 docker container on a amd64 debian bullseye host. The following steps reproduce it for me:

• create an amd64 Debian Bullseye VM in a cloud provider of your choice (we've seen this in both EC2 and GCP)
• install necessary packages for running arm64 docker images: sudo apt install libc6-arm64-cross qemu-user-static docker.io
• create an empty Go executable:

  package main
  func main() {}

• run it with sudo docker run --rm --platform linux/arm64/v8 -v $(pwd)/test.go:/go/test.go golang:1.20.1 go run test.go

This works with Go 1.19.4, but fails with Go 1.20.0 and 1.20.1:

$ sudo docker run --rm --platform linux/arm64/v8 -v $(pwd)/test.go:/go/test.go golang:1.20.1 go run test.go
  runtime: copying /usr/local/go/src/runtime/tls_arm64.h to /tmp/go-build4200235085/b002/tls_GOARCH.h: write /tmp/go-build4200235085/b002/tls_GOARCH.h: copy_file_range: function not implemented

$ sudo docker run --rm -ti --platform linux/arm64/v8 -v $(pwd)/test.go:/go/test.go golang:1.20.0 go run test.go
runtime: copying /usr/local/go/src/runtime/tls_arm64.h to /tmp/go-build4014913792/b002/tls_GOARCH.h: write /tmp/go-build4014913792/b002/tls_GOARCH.h: copy_file_range: function not implemented

$ sudo docker run --rm --platform linux/arm64/v8 -v $(pwd)/test.go:/go/test.go golang:1.19.4 go run test.go
<no output>

I was able to remove docker from the picture. Running directly with qemu results in the same behaviour:

$ qemu-aarch64-static -L /usr/aarch64-linux-gnu/ arm64/go/bin/go version
go version go1.20.1 linux/arm64

$ qemu-aarch64-static -L /usr/aarch64-linux-gnu/ arm64/go/bin/go run test.go 
runtime: copying /home/christopher_taylor/arm64/go/src/runtime/tls_arm64.h to /tmp/go-build1029963439/b002/tls_GOARCH.h: write /tmp/go-build1029963439/b002/tls_GOARCH.h: copy_file_range: function not implemented

I think this is interesting because similar to @m1d1, there's a virtualization layer involved. I don't know if VSERVER is qemu-based?

For completeness, to see if this is something to do with qemu, I tried running the same in qemu-x86_64-static, but only got segfaults, so I didn't look further.

$ amd64/go/bin/go version
go version go1.20.1 linux/amd64

$ qemu-x86_64-static amd64/go/bin/go version
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault

@ccmtaylor:

┌─[root]
└─▪ virt-what 
openvz
lxc

Can people who are having this problem please see if https://go.dev/cl/470016 fixes it. Thanks.

Change https://go.dev/cl/470016 mentions this issue: Revert "internal/poll: drop redundant ENOSYS in CopyFileRange"

@gopherbot Please open a backport to 1.20.

On some systems copy_file_range appears to fail with ENOSYS, even though it seems that it shouldn't. On those systems the go tool doesn't work at all.

Backport issue(s) opened: #58627 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

Can people who are having this problem please see if https://go.dev/cl/470016 fixes it. Thanks.

Also, could you guys try to run the example C code of copy_file_range in man page to see if your OS with kernel version > 5.3 surely get ENOSYS when calling copy_file_range? If so, I believe we should also file a bug report for Linux kernel?

@ianlancetaylor @panjf2000 I'll give those a try today

I can confirm that https://go.dev/cl/470016 fixes the issue for us. I cherry-picked the CL onto the go1.20.1 tag in a fresh golang/go checkout, buit it with CGO_ENABLED=0 GOROOT_BOOTSTRAP=~/amd64/go GOARCH=arm64 ./make.bash and ran these to confirm:

$ qemu-aarch64-static -L /usr/aarch64-linux-gnu/ goroot/bin/linux_arm64/go version
go version go1.20.1 linux/arm64
$ qemu-aarch64-static -L /usr/aarch64-linux-gnu/ goroot/bin/linux_arm64/go run test.go 

(CGO_ENABLED=0 was because I didn't have a cross-compiler installed. Trying again now) I rebuilt Go without CGO_ENABLED=0 and re-ran the tests successfully. Sounds like we're good to go.

Can people who are having this problem please see if https://go.dev/cl/470016 fixes it. Thanks.

Also, could you guys try to run the example C code of copy_file_range in man page to see if your OS with kernel version > 5.3 surely get ENOSYS when calling copy_file_range? If so, I believe we should also file a bug report for Linux kernel?

@panjf2000 I just gave this a try, too. Looks like copy_file_range doesn't work in this setup (Debian Bullseye amd64, running arm64 binaries via qemu):

$ cat [sample code from https://man7.org/linux/man-pages/man2/copy_file_range.2.html#EXAMPLES] > copy_file_range_test.c

$  CC=aarch64-linux-gnu-gcc make copy_file_range_test
aarch64-linux-gnu-gcc     copy_file_range_test.c   -o copy_file_range_test

$ touch foo

$ qemu-aarch64-static -L /usr/aarch64-linux-gnu ./copy_file_range_test foo bar
copy_file_range: Function not implemented

I see that the copy_file_range implementation in qemu may return ENOSYS under some circumstances, so maybe that's what we're seeing?

@ccmtaylor Thank you for the effort, we now know the behavior of qemu, it does look a bit unusual.

On the other hand, I'm also interested in how the real machine (other than emulator and virtualizer) acts on copy_file_range, could you also run the C example code on your machine and observe what happens, thanks! @m1d1

sure! the copy_file_range sample code works fine running natively on the host (an e2-medium instance in europe-west1):

$ uname -a
Linux ccmtaylor-bullseye-test 5.10.0-21-cloud-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

$ CC=gcc make copy_file_range_test
gcc     copy_file_range_test.c   -o copy_file_range_test

$ ./copy_file_range_test foo bar

$ ls -1s foo bar
0 bar
0 foo

the copy_file_range sample code works fine running natively on the host (an e2-medium instance in europe-west1)

What about go1.20? does it also work with copy_file_range?

the copy_file_range sample code works fine running natively on the host (an e2-medium instance in europe-west1)

What about go1.20? does it also work with copy_file_range?

not sure what you're asking. Yes, the release binaries for Go 1.20.1 work fine on the host:

# release binaries
$ amd64/go/bin/go run test.go
Hello World

# go1.20.1 patched with https://go.dev/cl/470016
$ goroot/bin/go run test.go
Hello World

@ccmtaylor Thanks.

Change https://go.dev/cl/470316 mentions this issue: [release-branch.go1.20] Revert "internal/poll: drop redundant ENOSYS in CopyFileRange"