Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools/gopls: gopls opens too many files with "go mod tidy" #71698

Open
Handzo opened this issue Feb 12, 2025 · 7 comments
Open

x/tools/gopls: gopls opens too many files with "go mod tidy" #71698

Handzo opened this issue Feb 12, 2025 · 7 comments
Labels
gopls Issues related to the Go language server, gopls. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone
gopls/backlog

Comments

@Handzo
Copy link

Handzo commented Feb 12, 2025

gopls version

Build info
----------
golang.org/x/tools/gopls v0.17.1
    golang.org/x/tools/gopls@v0.17.1 h1:Mt/DSfnnSe3dyf6MH/dZZ0iww+viHNhAFc4rEYDiOAw=
    github.com/BurntSushi/toml@v1.4.1-0.20240526193622-a339e1f7089c h1:pxW6RcqyfI9/kWtOwnv/G+AzdKuy2ZrqINhenH4HyNs=
    github.com/google/go-cmp@v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
    golang.org/x/exp/typeparams@v0.0.0-20231108232855-2478ac86f678 h1:1P7xPZEwZMoBoz0Yze5Nx2/4pxj6nw9ZqHWXqP0iRgQ=
    golang.org/x/mod@v0.22.0 h1:D4nJWe9zXqHOmWqj4VMOJhvzj7bEZg4wEYa759z1pH4=
    golang.org/x/sync@v0.9.0 h1:fEo0HyrW1GIgZdpbhCRO0PkJajUS5H9IFUztCgEo2jQ=
    golang.org/x/telemetry@v0.0.0-20241106142447-58a1122356f5 h1:TCDqnvbBsFapViksHcHySl/sW4+rTGNIAoJJesHRuMM=
    golang.org/x/text@v0.20.0 h1:gK/Kv2otX8gz+wn7Rmb3vT96ZwuoxnQlY+HlJVj7Qug=
    golang.org/x/tools@v0.27.1-0.20241219162658-575221bfbda3 h1:kgwdasJRsdDWYgWcEgMF424DiXwwXHSb3V8xVTi//i8=
    golang.org/x/vuln@v1.0.4 h1:SP0mPeg2PmGCu03V+61EcQiOjmpri2XijexKdzv8Z1I=
    honnef.co/go/tools@v0.5.1 h1:4bH5o3b5ZULQ4UrBmP+63W9r7qIkqJClEA9ko5YKx+I=
    mvdan.cc/gofumpt@v0.7.0 h1:bg91ttqXmi9y2xawvkuMXyvAA/1ZGJqYAEGjXuP0JXU=
    mvdan.cc/xurls/v2@v2.5.0 h1:lyBNOm8Wo71UknhUs4QTFUNNMyxy2JEIaKKo0RWOh+8=
go: go1.23.6

go env

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/handzo/Library/Caches/go-build'
GOENV='/Users/handzo/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/handzo/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/handzo/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/handzo/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOVCS=''
GOVERSION='go1.22.5'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/handzo/work/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/c_/xr46l7b17tl1vx8ynq0wqmjdjbq0ht/T/go-build3884281298=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Opening project in emacs with lsp-mode.

What did you see happen?

in ~= 30 minutes can do nothing. Getting "Too many open files in system" on any operation.

What did you expect to see?

Everything works well

Editor and settings

directory filters: ["-directory1", "-directory2", ...],
file watch ignored directories: ["directory1", "directory2", ...]

Logs

$ lsof -n | awk '{print $1}' | uniq -c | sort -rn | head -n 1
11417 go

$ ps -ax | grep go
2975 ??         1:34.37 go mod tidy -modfile=/var/folders/c_/xr46l7b17tl1vx8ynq0wqmjdjbq0ht/T/gopls-tempmod2565361955/go.mod

$ lsof -p 2975 | wc -l
10124
@Handzo Handzo added gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository. labels Feb 12, 2025
@gopherbot gopherbot added this to the Unreleased milestone Feb 12, 2025
@gabyhelp
Copy link

Related Issues

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@gabyhelp gabyhelp added the ToolProposal Issues describing a requested change to a Go tool or command-line program. label Feb 12, 2025
@jba jba added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed ToolProposal Issues describing a requested change to a Go tool or command-line program. labels Feb 13, 2025
@adonovan
Copy link
Member

@samthanawalla @matloob Any idea why the go mod tidy command might have 11417 open files? If there's some parallel recursion over the file tree, we should impose a reasonable limit on parallel I/O.

@matloob
Copy link
Contributor

matloob commented Feb 13, 2025

Usually when we recurse over the file tree we use par.Queues that limit the number of operations to GOMAXPROCS, and the opened files for each operation should be closed at the end of those operations.

One thing that comes to mind that may or may not be the issue is the module index: we mmap files that index each of the modules, and each of the packages in the main/work module and we don't close those files for the lifetime of the go command so it's possible that in a very large project could open too many of those files? I'm curious how many packages go list all ./... in the root of the main/work module returns, and how many entries go list -m all returns?

@adonovan
Copy link
Member

Does the go command close the fd for each mapped file? If so then the mapping shouldn't continue to count against the fd limit.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/652435 mentions this issue: cmd/go/internal/modindex: clean modroot and pkgdir for openIndexPackage

@rsc
Copy link
Contributor

rsc commented Feb 26, 2025

If we close the fd but leave the mmap'ed region, I think that still counts as an open file for the kernel-wide limit even if not an open file descriptor for the process. So closing them would make lsof look better but probably not avoid "Too many open files in system".

It looks like you're on a Mac. On my Mac, sysctl -a | grep file shows kern.maxfiles: 122880 on one laptop and kern.maxfiles: 276480 on another. Both are well above the 11,000 you saw being used by the go command. Perhaps something else was running away creating many processes with only a few files each and the go command was an innocent bystander?

What is your kern.maxfiles?

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/653055 mentions this issue: cmd/go/internal/mmap: close file after mmap

gopherbot pushed a commit that referenced this issue Feb 28, 2025
@matloob
cmd/go/internal/modindex: clean modroot and pkgdir for openIndexPackage
6894904 
GetPackage is sometimes called with a modroot or pkgdir that ends with a
path separator, and sometimes without. Clean them before passing them to
openIndexPackage to deduplicate calls to mcache.Do and action entry
files.

This shouldn't affect #71698 but was discovered while debugging that
issue.

Change-Id: I6a7fa4de93f45801504abea11bd97f6c6577f296
Reviewed-on: https://go-review.googlesource.com/c/go/+/652435
Reviewed-by: Alan Donovan <adonovan@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
gopherbot pushed a commit that referenced this issue Mar 10, 2025
@matloob
cmd/go/internal/mmap: close file after mmap
2620cc1 
Closing the file after mmap will reduce the number of files associated
with the process. This will not likely help with #71698 but it doesn't
hurt to close the files and should simplify lsof output.

For #71698

Change-Id: I06a1bf91914afc7703783fe1a38d8bc5a6fb3d9d
Reviewed-on: https://go-review.googlesource.com/c/go/+/653055
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
gopls Issues related to the Go language server, gopls. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Milestone
gopls/backlog
Development

No branches or pull requests
7 participants
@rsc @Handzo @adonovan @gopherbot @matloob @jba @gabyhelp