path/filepath: Walk is not walking over all files over NFS #25127
Labels
FrozenDueToAge
NeedsInvestigation
Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Comments
|
Are you sure that the problem is in ( |
bcmills
added
WaitingForInfo
|
Have you tried the same program with Go 1.9? (Does it exhibit the same problem?) |
|
@bcmills Yes, I was using Go 1.9 and got the bug. I updated to Go 1.10 and the bug persists. |
|
@felipenoris This program takes a single argument which is the directory to read. Could you run it on your NFS directory and see whether it lists all the files? If it fails to list a file, is it consistent as to which file it fails to list? Thanks. package main
import (
"fmt"
"os"
"sort"
)
func main() {
f, err := os.Open(os.Args[1])
if err != nil {
panic(err)
}
names, err := f.Readdirnames(-1)
if err != nil {
panic(err)
}
f.Close()
sort.Strings(names)
for _, n := range names {
fmt.Println(n)
}
} |
|
@ianlancetaylor , yes, that causes the same problem and is consistent with the problem that happens with I compared with package main
import (
"fmt"
"os"
"os/exec"
"strings"
)
func main() {
f, err := os.Open(os.Args[1])
if err != nil {
panic(err)
}
names, err := f.Readdirnames(-1)
if err != nil {
panic(err)
}
f.Close()
/*
for _, s := range names {
fmt.Println(s)
}
*/
fmt.Printf("Readdirnames listed %d files.\n", len(names))
cmd := exec.Command("ls", "-l", os.Args[1])
output, err := cmd.Output()
if err != nil {
fmt.Println(err)
return
}
lines := strings.Split(string(output), "\n")
/*
for _, s := range lines {
fmt.Println(s)
}
*/
fmt.Printf("'ls -l' listed %d files.\n", len(lines)-2) // ls outputs 2 additional lines
}Output fnoro@dbecceed1666:~/area/fnoro/src/learngo$ ./readdir_vs_ls /area/DB/ARC_ETL
Readdirnames listed 11 files.
'ls -l' listed 11 files.
fnoro@dbecceed1666:~/area/fnoro/src/learngo$ ./readdir_vs_ls /area/DB/ARC_ETL/CSV
Readdirnames listed 1 files.
'ls -l' listed 1 files.
fnoro@dbecceed1666:~/area/fnoro/src/learngo$ ./readdir_vs_ls /area/DB/ARC_ETL/CSV/20180426
Readdirnames listed 82 files.
'ls -l' listed 83 files. |
|
cc @opm22, @accerqueira |
|
Is it always missing the same file? Can you tell us anything about that file, like its |
|
@ianlancetaylor , yes, at least for today. I guess yesterday it picked another file to miss. I made a slight modification to the test program as below, to narrow down the missing file, and to provide the package main
import (
"fmt"
"os"
"os/exec"
"strings"
)
func main() {
f, err := os.Open(os.Args[1])
if err != nil {
panic(err)
}
names, err := f.Readdirnames(-1)
if err != nil {
panic(err)
}
f.Close()
var readdir_swap_names []string
for _, s := range names {
// fmt.Println(s)
if strings.Contains(s, "swap") {
fmt.Println(s)
readdir_swap_names = append(readdir_swap_names, s)
}
}
//fmt.Printf("Readdirnames listed %d files.\n", len(names))
fmt.Printf("Readdirnames listed %d swap names.\n", len(readdir_swap_names))
cmd := exec.Command("ls", "-l", os.Args[1])
output, err := cmd.Output()
if err != nil {
fmt.Println(err)
return
}
lines := strings.Split(string(output), "\n")
var ls_swap_names []string
for _, s := range lines {
// fmt.Println(s)
if strings.Contains(s, "swap") {
fmt.Println(s)
ls_swap_names = append(ls_swap_names, s)
}
}
// fmt.Printf("'ls -l' listed %d files.\n", len(lines)-2)
fmt.Printf("'ls -l' listed %d swap names.\n", len(ls_swap_names))
}Output: There you can see that |
|
Thanks. Unfortunately I don't see anything informative there. Can you run your program as |
|
Yes, this is the output. Thanks @accerqueira for generating this log! |
bcmills
removed
the
WaitingForInfo
|
Thanks. The difference is that the Go code calls the Here are the calls from Go: What version of the kernel are you using? This may be a dup of #24015. |
|
@accerqueira, can you provide us information about the kernel version? |
|
Kernel version was 3.10.0-693.11.1.el7.x86_64 (Red Hat Enterprise Linux Server 7.4) Tried on 4.13.0-39-generic (Ubuntu 17.10) and it seems to work: With the following calls from Go: Looks like a getdents64 bug on that kernel version, right? |
|
@felipenoris is this really on NFS, or is it actually a CIFS mount? The duplicate bug is for a CIFS specific code path. |
|
@accerqueira , can you answer that? |
|
It's a CIFS mount |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?What did you do?
The function
filepath.Walkskips exactly one file when iterating over a NFS mapped directory of about 100 files.What did you expect to see?
My directory has exactly 83 files.
What did you see instead?
Only 82 files are listed. The skipped file is random.
The text was updated successfully, but these errors were encountered: