Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testing/fstest: fstest.TestFS is slow when given directory contains relatively big files #69888

Closed
soulim opened this issue Oct 15, 2024 · 2 comments
Closed

testing/fstest: fstest.TestFS is slow when given directory contains relatively big files #69888

soulim opened this issue Oct 15, 2024 · 2 comments

Comments

@soulim
Copy link

soulim commented Oct 15, 2024 •
edited
Loading

Go version

go version go1.23.2 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/asulim/Library/Caches/go-build'
GOENV='/Users/asulim/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/asulim/.local/share/asdf/installs/golang/1.23.2/packages/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/asulim/.local/share/asdf/installs/golang/1.23.2/packages'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/asulim/.local/share/asdf/installs/golang/1.23.2/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/asulim/.local/share/asdf/installs/golang/1.23.2/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.23.2'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/asulim/Library/Application Support/go/telemetry'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/asulim/data/sources/github.com/soulim/testfstest/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/wf/vzjbb9654pd88ndz56k77xqh0000gn/T/go-build734442337=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

A call to fstest.TestFS takes significant time (seconds) to check a directory, when it contains at least one relatively big file.

I've created a demo repository with a set of tests relying on fstest.TestFS: https://github.com/soulim/testfstest. Additionally there's a GitHub workflow that runs the same tests to demonstrate results.

Setup

  • testdata/many-big directory contains a set of files and one of them is big (4.4MB).
  • testdata/many-small directory contains a set of file, but none of them is big.
  • testdata/one-<category> directories contain only one file of each category: big, small, and empty.

Each time when a test works with a directory containing a big file, it takes few seconds for fstest.TestFS to return results.

What did you see happen?

An example test with fstest.TestFS:

func TestTestFS(t *testing.T) {
	files := []string{
		filepath.Join("testdata", "many-big", "big.jpg"),
		filepath.Join("testdata", "many-big", "empty.txt"),
		filepath.Join("testdata", "many-big", "small.txt"),
		filepath.Join("testdata", "many-small", "empty.txt"),
		filepath.Join("testdata", "many-small", "small.txt"),
		filepath.Join("testdata", "one-big", "big.jpg"),
		filepath.Join("testdata", "one-empty", "empty.txt"),
		filepath.Join("testdata", "one-small", "small.txt"),
	}

	for _, file := range files {
		t.Run(file, func(t *testing.T) {
			var ts0, ts1 time.Time

			dir := filepath.Dir(file)

			ts0 = time.Now()
			err := fstest.TestFS(os.DirFS(dir), filepath.Base(file))
			ts1 = time.Now()
			t.Logf("TestFS() time: %v", ts1.Sub(ts0))
			if err != nil {
				t.Fatal(err)
			}
		})
	}
}

Output:

go test ./... -v
=== RUN   TestTestFS
=== RUN   TestTestFS/testdata/many-big/big.jpg
    main_test.go:32: TestFS() time: 4.008915917s
=== RUN   TestTestFS/testdata/many-big/empty.txt
    main_test.go:32: TestFS() time: 3.955715375s
=== RUN   TestTestFS/testdata/many-big/small.txt
    main_test.go:32: TestFS() time: 3.953422583s
=== RUN   TestTestFS/testdata/many-small/empty.txt
    main_test.go:32: TestFS() time: 2.408916ms
=== RUN   TestTestFS/testdata/many-small/small.txt
    main_test.go:32: TestFS() time: 721.459µs
=== RUN   TestTestFS/testdata/one-big/big.jpg
    main_test.go:32: TestFS() time: 3.968942875s
=== RUN   TestTestFS/testdata/one-empty/empty.txt
    main_test.go:32: TestFS() time: 243.958µs
=== RUN   TestTestFS/testdata/one-small/small.txt
    main_test.go:32: TestFS() time: 2.164458ms
--- PASS: TestTestFS (15.89s)
    --- PASS: TestTestFS/testdata/many-big/big.jpg (4.01s)
    --- PASS: TestTestFS/testdata/many-big/empty.txt (3.96s)
    --- PASS: TestTestFS/testdata/many-big/small.txt (3.95s)
    --- PASS: TestTestFS/testdata/many-small/empty.txt (0.00s)
    --- PASS: TestTestFS/testdata/many-small/small.txt (0.00s)
    --- PASS: TestTestFS/testdata/one-big/big.jpg (3.97s)
    --- PASS: TestTestFS/testdata/one-empty/empty.txt (0.00s)
    --- PASS: TestTestFS/testdata/one-small/small.txt (0.00s)
PASS
ok      github.com/soulim/testfstest    16.170s

As you could see above it took 15 seconds for the test suite to finish. Each test is only using fstest.TestFS.

What did you expect to see?

I did check the documentation for fstest.TestFS and haven't found any warning regarding working with big files.

fstest.TestFS does many checks and maybe it's expected for it to be slow with big files. However in that case, it would be nice to inform about that in the documentation.

I'm more than happy to provide any additional details.

Thank you a lot for being awesome!
@gabyhelp
Copy link

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@seankhliao
Copy link
Member

I think you just have a slow disk

$ go test -v                   
=== RUN   TestTestFS
=== RUN   TestTestFS/testdata/many-big/big.jpg
    main_test.go:32: TestFS() time: 1.640587907s
=== RUN   TestTestFS/testdata/many-big/empty.txt
    main_test.go:32: TestFS() time: 1.63107534s
=== RUN   TestTestFS/testdata/many-big/small.txt
    main_test.go:32: TestFS() time: 1.639283998s
=== RUN   TestTestFS/testdata/many-small/empty.txt
    main_test.go:32: TestFS() time: 252.364µs
=== RUN   TestTestFS/testdata/many-small/small.txt
    main_test.go:32: TestFS() time: 231.484µs
=== RUN   TestTestFS/testdata/one-big/big.jpg
    main_test.go:32: TestFS() time: 1.63203552s
=== RUN   TestTestFS/testdata/one-empty/empty.txt
    main_test.go:32: TestFS() time: 76.573µs
=== RUN   TestTestFS/testdata/one-small/small.txt
    main_test.go:32: TestFS() time: 202.798µs
--- PASS: TestTestFS (6.54s)
    --- PASS: TestTestFS/testdata/many-big/big.jpg (1.64s)
    --- PASS: TestTestFS/testdata/many-big/empty.txt (1.63s)
    --- PASS: TestTestFS/testdata/many-big/small.txt (1.64s)
    --- PASS: TestTestFS/testdata/many-small/empty.txt (0.00s)
    --- PASS: TestTestFS/testdata/many-small/small.txt (0.00s)
    --- PASS: TestTestFS/testdata/one-big/big.jpg (1.63s)
    --- PASS: TestTestFS/testdata/one-empty/empty.txt (0.00s)
    --- PASS: TestTestFS/testdata/one-small/small.txt (0.00s)
PASS
ok  	github.com/soulim/testfstest	6.547s

The docs already say:

It walks the entire tree of files in fsys, opening and checking that each file behaves correctly.

With it checking every file, it's fast if your filesystem is fast (e.g. MapFS), and slow if you have a slow filesystem. If you tested a directory of many small files that add up to the same size, it's about the same speed. I don't think any special warning is necessary for large files.

@seankhliao seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@soulim @seankhliao @gabyhelp