Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls: high memory consumption #47855

Closed
orlovm opened this issue Aug 20, 2021 · 16 comments
Closed

x/tools/gopls: high memory consumption #47855

orlovm opened this issue Aug 20, 2021 · 16 comments
Assignees
Labels
gopls/performance Issues related to gopls performance (CPU, memory, etc). gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone

Comments

@orlovm
Copy link

orlovm commented Aug 20, 2021

What version of Go are you using (go version)?

$ go version
go version go1.17 linux/amd64

Does this issue reproduce with the latest release?

Yes. I have the latest versions of gopls and go.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mikhail/.cache/go-build"
GOENV="/home/mikhail/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/mikhail/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mikhail/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4125386380=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Opened cockroachDB repository in vim(same thing with neovim) with vim-go plugin and gopls enabled.
I've removed everything but vim-go and gopls from the .vimrc and disabled all other plugins. But the result is the same.

What did you expect to see?

Normal memory consumption. For example, Jetbrains Goland uses ~2GB when this project is open

What did you see instead?

gopls consumes about 9GB of memory, sometimes more than 14GB.

gopls.30806-8GiB-withnames.zip

gopls.3081-14GiB-withnames.zip

@gopherbot gopherbot added Tools This label describes issues relating to any tools in the x/tools repository. gopls Issues related to the Go language server, gopls. labels Aug 20, 2021
@gopherbot gopherbot added this to the Unreleased milestone Aug 20, 2021
@heschi
Copy link
Contributor

heschi commented Aug 25, 2021

In general gopls can be fairly memory hungry on large projects. This is an ongoing area of research. cc @findleyr.

FWIW, glancing at the attached zip, github.com/cockroachdb/cockroach/pkg/workload/geospatial is probably costing you 500+ MB all by itself. That package contains a very large composite literal that is very expensive to parse and type check. Removing that composite literal from gopls' scope, perhaps by changing its representation or excluding it from the build with a build tag, might be well worth the effort.

@orlovm
Copy link
Author

orlovm commented Aug 25, 2021

@heschi I've added a new archive. Please take a look at it.
gopls consumes more memory during use.
Isn't there any memory leak?

@heschi
Copy link
Contributor

heschi commented Aug 25, 2021

No. I don't see any signs of a memory leak.

If you want, you can try setting https://github.com/golang/tools/blob/master/gopls/doc/settings.md#memorymode-enum. Use it at your own risk.

@orlovm
Copy link
Author

orlovm commented Aug 26, 2021

So this is normal behavior?
Why is there such a big difference in memory consumption compared to JB Goland?

@hyangah
Copy link
Contributor

hyangah commented Aug 27, 2021

I'd say the high memory usage is expected currently - Goland and gopls take different approaches.

Gopls dev team is currently working on improvement in this area. The "DegradeClosed" memory mode @heschi shared is one of the directions we are exploring. Please give it a try (warning: still experimental). Feedback and bug reports for the DegradeClosed mode are greatly appreciated!

@findleyr findleyr added gopls/performance Issues related to gopls performance (CPU, memory, etc). and removed Performance labels Oct 8, 2021
@findleyr findleyr removed their assignment Aug 9, 2022
@findleyr findleyr modified the milestones: gopls/unplanned, gopls/later Aug 9, 2022
@karlismea
Copy link

karlismea commented Sep 21, 2022

In general gopls can be fairly memory hungry on large projects.

I thought all source needs to be in /src directory - it means hundreds of imports and all my projects! Happy if someone can explains how to remove this "virus" from VS Code.

@adonovan adonovan self-assigned this Dec 8, 2022
@578559967
Copy link

How to enable "DegradeClosed" memory mode in vim?

I'd say the high memory usage is expected currently - Goland and gopls take different approaches.

Gopls dev team is currently working on improvement in this area. The "DegradeClosed" memory mode @heschi shared is one of the directions we are exploring. Please give it a try (warning: still experimental). Feedback and bug reports for the DegradeClosed mode are greatly appreciated!

@adonovan
Copy link
Member

The next release of gopls (expected in March) will include major reductions in memory usage, so if you're on the verge of giving up, consider holding on just a few more weeks.

@n-r-w
Copy link

n-r-w commented Mar 27, 2023

image

@findleyr
Copy link
Contributor

@n-r-w which version of gopls are you using?

We're still smoothing out some edge cases and so have not yet cut a prerelease for gopls@v0.12.0, but I expect memory usage to be significantly reduced once we do.

@n-r-w
Copy link

n-r-w commented Mar 27, 2023

@n-r-w which version of gopls are you using?

v0.11.0

@n-r-w
Copy link

n-r-w commented Mar 27, 2023

We're still smoothing out some edge cases and so have not yet cut a prerelease for gopls@v0.12.0, but I expect memory usage to be significantly reduced once we do.

It's great to hear that. Because right now my gopls is using 34 Gb of memory :)
image

@findleyr findleyr removed this from the gopls/later milestone Apr 3, 2023
@findleyr findleyr added this to the gopls/v0.12.0 milestone Apr 3, 2023
@findleyr
Copy link
Contributor

findleyr commented Apr 25, 2023

Hi all, we just cut the first prerelease version of gopls to include the new architecture:

go install golang.org/x/tools/gopls@v0.12.0-pre.1

More details in the release notes.

This is a prerelease version: there are several known regressions that we're still working on. However, it should use significantly less memory. I'd love to hear from users currently struggling with gopls' memory usage, whether this release improves their experience. If it doesn't, there may be more we can do given additional details.

Thanks!

@garymoon
Copy link

Early impressions are very positive for me: Working with the Gitea codebase in VSCode, after letting things settle, my gopls RSS has almost halved, from 1,044,976 to 568,124, and in the time it's taken me to write this has somehow fallen further to 478,228. You've made a cranky old infra engineer very happy, thank you all very much! 💙

@findleyr
Copy link
Contributor

findleyr commented Apr 26, 2023

Thanks for the feedback, @garymoon!

in the time it's taken me to write this has somehow fallen further to 478,228

FWIW this is likely due to a background GC cycle, and/or the timing of background work that gopls is doing. To get a more detailed picture of memory usage, you can start gopls with -debug=localhost:8080 and visit http://localhost:8080/memory.

@findleyr
Copy link
Contributor

Hi folks,

We just released the second prerelease, which fixes several bugs reported since -pre.1 and smooths out performance a bit:

go install golang.org/x/tools/gopls@v0.12.0-pre.2

At this point, gopls' memory footprint has fundamentally changed: gopls requires very little data in memory other than file content. To reduce its steady-state memory further would require dropping file content, and we don't current plan to do that (doing so would introduce a lot of complexity). There may be additional issues related to memory: for example, gopls could use less memory during large operations such as rename. However, those would be specific problems that deserve a separate github issue.

Therefore, I am going to close this issue as fixed. Thanks everyone for your feedback and patience.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
gopls/performance Issues related to gopls performance (CPU, memory, etc). gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Development

No branches or pull requests

10 participants