os: Readdirnames throws errors on CIFS #39237

rfjakob · 2020-05-24T22:11:51Z

What version of Go are you using (`go version`)?

$ go version
go version go1.14.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jakob/.cache/go-build"
GOENV="/home/jakob/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jakob/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/jakob/go/src/github.com/rfjakob/gocryptfs/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build655782218=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Run readdirnames.go against a cifs mount:

while sleep 1 ; do ./readdirnames /mnt/synology/public/tmp/g1 ; done

Mount info:

//192.168.0.3/public on /mnt/synology/public type cifs (rw,relatime,vers=3.1.1,cache=strict,username=jakob,uid=1026,forceuid,gid=1026,forcegid,addr=192.168.0.3,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)

Linux kernel:

5.6.13-300.fc32.x86_64

What did you expect to see?

No errors, consistent results, like from the C program getdents.c

$ while sleep 1 ; do ./getdents_c /mnt/synology/public/tmp/g1 ; done
getdents64 fd3: n=4176, errno=0
getdents64 fd3: n=4176, errno=0
getdents64 fd3: n=4176, errno=0
getdents64 fd3: n=4176, errno=0
getdents64 fd3: n=4176, errno=0
getdents64 fd3: n=3192, errno=0
getdents64 fd3: n=0, errno=0
total 24072 bytes

[repeats ad infinitum]

What did you see instead?

readdirnames.go :

$ while sleep 1 ; do ./readdirnames /mnt/synology/public/tmp/g1 ; done
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=694, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=346, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=346, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=346, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=520, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=694, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
2020/05/24 23:59:25 os.Open returned err=open /mnt/synology/public/tmp/g1: interrupted system call
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>

The problem is easier to trigger when the terminal window is continously resized (this triggers SIGWINCH signals).

This is problably the same underlying issue as in #24015 , and I agree that it looks like a kernel bug.

However, the C program seems to handle this fine, even with window resizing stress. So if glibc has workarounds for this, maybe Go should have them as well.

Downstream issue: rfjakob/gocryptfs#483

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2020-05-25T00:08:42Z

This seems like the same issue as #38836. Is it possible for you to check whether the problem is fixed on master? Thanks.

rfjakob · 2020-05-25T06:23:15Z

Yes can do. Looks the same on master:

$ go build
$ go version
go version devel +f65ad0dda7 Sun May 24 20:49:55 2020 +0000 linux/amd64
$ while sleep 1 ; do ./readdirnames /mnt/synology/public/tmp/g1 ; done
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
2020/05/25 08:20:36 os.Open returned err=open /mnt/synology/public/tmp/g1: interrupted system call
Readdirnames: len=172, err=readdirent: no such file or directory
Readdirnames: len=868, err=readdirent: no such file or directory
Readdirnames: len=346, err=readdirent: no such file or directory
Readdirnames: len=346, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
^C

rfjakob · 2020-06-07T10:44:11Z

I posted a summary of the issue to linux-cifs and linux-kernel a few day ago:
https://lore.kernel.org/linux-cifs/2d735cb666872099b2c4c574e35aaeac2f7356ad.camel@gmail.com/

No reply yet, but I'm pretty sure now that this is a kernel bug. Does it make sense to keep this ticket open for tracking the issue?

networkimprov · 2020-06-07T15:13:54Z

@gopherbot remove WaitingForInfo

gopherbot · 2020-06-08T21:36:54Z

Change https://golang.org/cl/236997 mentions this issue: os: always check for EINTR in calls to open

ianlancetaylor · 2020-06-08T21:36:54Z

I'm going to add a check for EINTR in calls to open. I think that should fix the problem with Go.

rfjakob · 2020-06-08T22:02:30Z

Nice, that should fix the open problem. The ENOENT in the middle of readdirent is tricky, though.

Readdirnames: len=868, err=readdirent: no such file or directory

For #11180 For #20400 For #39237 Change-Id: I8de97517c8f92c08f0c8a51f651a17e31617979b Reviewed-on: https://go-review.googlesource.com/c/go/+/236997 Reviewed-by: Bryan C. Mills <bcmills@google.com>

ianlancetaylor · 2020-06-08T22:08:39Z

Quite right, sorry, it will only fix the problem with os.Open. I don't know what to do with the readdirent problem, but it sounds like that is a kernel bug, and not something we can avoid in Go. So I'm going to close this issue now. Please comment if you disagree.

For golang#11180 For golang#20400 For golang#39237 Change-Id: I8de97517c8f92c08f0c8a51f651a17e31617979b Reviewed-on: https://go-review.googlesource.com/c/go/+/236997 Reviewed-by: Bryan C. Mills <bcmills@google.com> (cherry picked from commit e64675a)

ncw · 2020-07-29T15:38:04Z

Rclone users have been having the same problem (see forum)

Using go1.15rc1 appears to fix the EINTR errors but does not fix the ENOENT errors.

Using export GODEBUG=asyncpreemptoff=1 does fix the problem completely.

So I guess CIFS is incorrectly returning ENOENT instead of EINTR on receipt of a signal?

@rfjakob did you get anywhere with trying to report this to the CIFS developers or upstream? I don't see any replies to your message.

rfjakob · 2020-07-29T18:39:31Z

Hi, no, I did not get any response to my posting, unfortunately. I don't really know what going on inside CIFS, but from the debug log I got out of the kernel, this line looks like something goes wrong internally:

[168517.187072] Status code returned 0x80000006 STATUS_NO_MORE_FILES

I don't understand why this does not happen with a C program. I can hammer getdents.c with SIGWINCH signals as much as I want, but I don't get ENOENT errors.

GODEBUG=asyncpreemptoff=1 also seems to fix it here, thanks for the tip!

networkimprov · 2020-07-29T19:06:00Z

hammer getdents.c with SIGWINCH signals

The Go runtime uses SIGURG (I believe), so perhaps that's what's tripping CIFS?

ianlancetaylor · 2020-07-30T03:37:47Z

I don't understand why this does not happen with a C program. I can hammer getdents.c with SIGWINCH signals as much as I want, but I don't get ENOENT errors.

Note that an EINTR error only occurs if a signal handler is installed using the sigaction system call.

meetme2meat · 2021-03-08T11:44:54Z

We are often seeing this happening in our system when closing an open File located in mounted Azure File Share(which I believe is CISF system). Our go version is 1.15.x which according this kubernetes/kubernetes#92521 would have the fixed for it.

Now, I confuse whether to use export GODEBUG=asyncpreemptoff=1 or not.

networkimprov · 2021-03-08T17:47:57Z

@meetme2meat, can you create a new issue describing your problem, with directions to reproduce it and the error results that you're seeing?

ianlancetaylor added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 25, 2020

odeke-em changed the title ~~os.Readdirnames throws errors on CIFS~~ os: Readdirnames throws errors on CIFS May 26, 2020

gopherbot removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 7, 2020

toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 8, 2020

toothrot added this to the Backlog milestone Jun 8, 2020

ianlancetaylor closed this as completed Jun 8, 2020

ianlancetaylor mentioned this issue Jul 9, 2020

internal/poll: can get EINTR reading from CIFS #38836

Closed

networkimprov mentioned this issue Aug 19, 2020

os: Missed checking for EINTR in (*os.File).readdir #40846

Closed

networkimprov mentioned this issue Aug 26, 2020

proposal: runtime: allow termination of blocked syscalls #41054

Closed

networkimprov mentioned this issue Sep 3, 2020

proposal: cmd/go: build option to set env variable defaults #40870

Closed

greatroar mentioned this issue Oct 2, 2020

Restic 0.10.0 causes read errors on mounted samba share restic/restic#2968

Closed

MichaelEischer mentioned this issue Oct 7, 2020

restic fails on repo mounted via CIFS/samba on Linux using go 1.14 build restic/restic#2659

Open

rfjakob mentioned this issue Oct 13, 2020

CIFS throws EINTR and ENOENT errors rfjakob/gocryptfs#483

Closed

jd1 mentioned this issue Nov 18, 2020

Library: Readdirnames throws errors on CIFS photoprism/photoprism#615

Closed

drakkan mentioned this issue Dec 9, 2020

If the directory contains more than ~3000 files, sftp client reported an error drakkan/sftpgo#243

Closed

kzc mentioned this issue Apr 2, 2021

Racey Could not resolve file errors despite files existing evanw/esbuild#1101

Closed

ncw mentioned this issue Apr 26, 2021

Rclone over Samba: very slow transfer speeds & network errors rclone/rclone#5163

Closed

muCruiser mentioned this issue May 4, 2021

Problem with cifs share - Could not read directory photoview/photoview#361

Closed

golang locked and limited conversation to collaborators Mar 8, 2022

gopherbot added the FrozenDueToAge label Mar 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os: Readdirnames throws errors on CIFS #39237

os: Readdirnames throws errors on CIFS #39237

rfjakob commented May 24, 2020 •

edited

ianlancetaylor commented May 25, 2020

rfjakob commented May 25, 2020

rfjakob commented Jun 7, 2020

networkimprov commented Jun 7, 2020

gopherbot commented Jun 8, 2020

ianlancetaylor commented Jun 8, 2020

rfjakob commented Jun 8, 2020

ianlancetaylor commented Jun 8, 2020

ncw commented Jul 29, 2020

rfjakob commented Jul 29, 2020

networkimprov commented Jul 29, 2020

ianlancetaylor commented Jul 30, 2020

meetme2meat commented Mar 8, 2021 •

edited

networkimprov commented Mar 8, 2021

os: Readdirnames throws errors on CIFS #39237

os: Readdirnames throws errors on CIFS #39237

Comments

rfjakob commented May 24, 2020 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

ianlancetaylor commented May 25, 2020

rfjakob commented May 25, 2020

rfjakob commented Jun 7, 2020

networkimprov commented Jun 7, 2020

gopherbot commented Jun 8, 2020

ianlancetaylor commented Jun 8, 2020

rfjakob commented Jun 8, 2020

ianlancetaylor commented Jun 8, 2020

ncw commented Jul 29, 2020

rfjakob commented Jul 29, 2020

networkimprov commented Jul 29, 2020

ianlancetaylor commented Jul 30, 2020

meetme2meat commented Mar 8, 2021 • edited

networkimprov commented Mar 8, 2021

rfjakob commented May 24, 2020 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

meetme2meat commented Mar 8, 2021 •

edited