Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

os: File.Readdirnames order is different than of "ls -l" #43435

Closed
dxps opened this issue Dec 30, 2020 · 6 comments
Closed

os: File.Readdirnames order is different than of "ls -l" #43435

dxps opened this issue Dec 30, 2020 · 6 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@dxps
Copy link

dxps commented Dec 30, 2020 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.15.6 linux/amd64

Does this issue reproduce with the latest release?

Yes, this is the latest release at the time of this writing.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/devisions/.cache/go-build"
GOENV="/home/devisions/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/devisions/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/devisions/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/devisions/apps/go-1.15.6"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/devisions/apps/go-1.15.6/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build624368339=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Since I want to find a more performant alternative to ioutil.ReadDir (that takes around 1.2 seconds if there are over 100.000 files in the directory), this os.File.Readdirnames seems to be the perfect option.

But the order of entries that are provided does not seem to be consistent the same as of ls -l. Of course, some ordering can be done on the result, but the whole point was to quickly get the next couple of files from a directory, sorted either by name or creation time.

The API documentation mentions that:

Readdirnames reads the contents of the directory associated with file and returns a slice of up to n names of files in the directory, in directory order.

Now, what in directory order means is yet unclear to me.

The code used is extremely simple:

f, err := os.Open(path)
defer func() { _ = f.Close() }()
if err != nil {
	return "", err
}
fns, err := f.Readdirnames(100)
if err != nil {
	return "", err
}
log.Printf("[dbg] Readdirnames => %v\n", fns)

What did you expect to see?

I was expected to see the same output as of ls:

1609349600755907389.dat  1609349614877620779.dat  1609349614878350488.dat  1609349615380619680.dat

What did you see instead?

But - a concrete output, related to the same content as previously listed - the result of running the code is:

[dbg] Readdirnames => [1609349615380619680.dat 1609349600755907389.dat 1609349614877620779.dat 1609349614878350488.dat]

Any advice, please?

Thank you!
@dmitshur dmitshur changed the title os.File.Readdirnames unpredictable order os: File.Readdirnames unpredictable order Dec 30, 2020
@dxps dxps changed the title os: File.Readdirnames unpredictable order os.File.Readdirnames order is different than of "ls -l" Dec 30, 2020
@dmitshur
Copy link
Contributor

Directory order means the same order that the operating system returns. This depends on the OS type, version, and importantly the filesystem used at the path that was read.

To get a predictable order across systems, you need to perform sorting yourself, as done by ioutil.ReadDir.

I think this is working as intended and documented. Please comment if I missed something.

@dmitshur dmitshur changed the title os.File.Readdirnames order is different than of "ls -l" os: File.Readdirnames order is different than of "ls -l" Dec 30, 2020
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 30, 2020
@dmitshur dmitshur added this to the Backlog milestone Dec 30, 2020
@dxps
Copy link
Author

dxps commented Dec 30, 2020

@dmitshur Thanks for the feedback.

brb with the result.

@dmitshur
Copy link
Contributor

I'll close this as there's no bug here, and we use the Go issue tracker for bugs and proposals only.

For questions and discussion about using Go, please see https://golang.org/wiki/Questions for a list of better places. Thanks.

P.S. In your snippet, please note that you should handle the error from os.Open first, and call defer f.Close() second, not vice versa. Otherwise, the code may call Close on an invalid file.

@dxps
Copy link
Author

dxps commented Dec 30, 2020

As promised, got back with the result. I was about to close this issue anyway... 😊

As a summary, I am happy with these figures:

  • although sorting is needed (basically, used fnames, err = f.Readdirnames(0) then sort.Strings(fnames))
  • the overall spent time is considerable smaller than of ioutil.ReadDir's one.
>>> ioutil.ReadDir exec time: 1.311758808s
>>> os.File.Readdirnames exec time: 144.131977ms
>>> os.File.Readdirnames result has 499099 entries.
>>> sort exec time: 163.322515ms

P.S. You're right, thanks! The code was just for this sample purposes.

@dmitshur
Copy link
Contributor

Yep, the time difference is expected because ioutil.ReadDir calls f.Readdir which performs lstat for each file to get more information (os.FileInfo), while f.Readdirnames(0) returns only the file names.

Note that starting with Go 1.16, there will be a new API os.ReadDir that you might find more convenient to use. It's fast by default, but lets you get more information on demand.

@dxps
Copy link
Author

dxps commented Dec 30, 2020

Yes, read about the upcoming API. Thanks.

@golang golang locked and limited conversation to collaborators Dec 30, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
3 participants
@dmitshur @gopherbot @dxps