Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools/gopls: high memory consumption #54268

Closed
codisms opened this issue Aug 4, 2022 · 7 comments
Closed

x/tools/gopls: high memory consumption #54268

codisms opened this issue Aug 4, 2022 · 7 comments
Labels
gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone
gopls/v0.12.0

Comments

@codisms
Copy link

codisms commented Aug 4, 2022

gopls version

Build info
----------
golang.org/x/tools/gopls v0.9.1
    golang.org/x/tools/gopls@v0.9.1 h1:SigsTL4Hpv3a6b/a7oPCLRv5QUeSM6PZNdta1oKY4B0=
    github.com/google/go-cmp@v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/mod@v0.6.0-dev.0.20220419223038-86c51ed26bb4 h1:6zppjxzCulZykYSLyVDYbneBfbaBIQPYMevg0bEwv2s=
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
    golang.org/x/sys@v0.0.0-20220209214540-3681064d5158 h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c=
    golang.org/x/text@v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
    golang.org/x/tools@v0.1.12-0.20220713141851-7464a5a40219 h1:Ljlba2fVWOA1049JjsKii44g8nZN2GjpxMlzVc8AnQM=
    mvdan.cc/gofumpt@v0.3.0 h1:kTojdZo9AcEYbQYhGuLf/zszYthRdhDNDUi2JKTxas4=
    mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: go1.16.15

go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOENV="/home/ubuntu/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ubuntu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ubuntu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.15"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3575468840=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Editing Go files with vim and vim-go.

What did you expect to see?

Lower memory consumption.

What did you see instead?

Over 5GB of memory consumption; thrashing swap file.

Editor and settings

VIM - Vi IMproved 9.0 (2022 Jun 28, compiled Aug  1 2022 05:17:14)
Included patches: 1-124
Compiled by ubuntu@dev-veritone
Huge version without GUI.  Features included (+) or not (-):
+acl               +file_in_path      +mouse_urxvt       -tag_any_white
+arabic            +find_in_path      +mouse_xterm       -tcl
+autocmd           +float             +multi_byte        +termguicolors
+autochdir         +folding           +multi_lang        +terminal
-autoservername    -footer            -mzscheme          +terminfo
-balloon_eval      +fork()            +netbeans_intg     +termresponse
+balloon_eval_term +gettext           +num64             +textobjects
-browse            -hangul_input      +packages          +textprop
++builtin_terms    +iconv             +path_extra        +timers
+byte_offset       +insert_expand     +perl/dyn          +title
+channel           +ipv6              +persistent_undo   -toolbar
+cindent           +job               +popupwin          +user_commands
-clientserver      +jumplist          +postscript        +vartabs
-clipboard         +keymap            +printer           +vertsplit
+cmdline_compl     +lambda            +profile           +vim9script
+cmdline_hist      +langmap           -python            +viminfo
+cmdline_info      +libcall           +python3/dyn       +virtualedit
+comments          +linebreak         +quickfix          +visual
+conceal           +lispindent        +reltime           +visualextra
+cryptv            +listcmds          +rightleft         +vreplace
+cscope            +localmap          +ruby/dyn          +wildignore
+cursorbind        -lua               +scrollbind        +wildmenu
+cursorshape       +menu              +signs             +windows
+dialog_con        +mksession         +smartindent       +writebackup
+diff              +modify_fname      -sodium            -X11
+digraphs          +mouse             -sound             -xfontset
-dnd               -mouseshape        +spell             -xim
-ebcdic            +mouse_dec         +startuptime       -xpm
+emacs_tags        -mouse_gpm         +statusline        -xsmp
+eval              -mouse_jsbterm     -sun_workshop      -xterm_clipboard
+ex_extra          +mouse_netterm     +syntax            -xterm_save
+extra_search      +mouse_sgr         +tag_binary
-farsi             -mouse_sysmouse    -tag_old_static
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -g -O2 -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc -L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-E -L/usr/local/lib -Wl,--as-needed -o vim -lm -ltinfo -lrt -ldl -Wl,-E -fstack-protector-strong -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu/perl/5.30/CORE -lperl -ldl -lm -lpthread -lcrypt

Logs

gopls.3782930-5GiB-nonames.zip
@gopherbot gopherbot added Tools This label describes issues relating to any tools in the x/tools repository. gopls Issues related to the Go language server, gopls. labels Aug 4, 2022
@gopherbot gopherbot added this to the Unreleased milestone Aug 4, 2022
@findleyr
Copy link
Contributor

findleyr commented Aug 5, 2022

That gopls uses a lot of memory is a known (and difficult) problem. See for example #47855.

However, 5gb is a lot, and we should check whether that makes sense based on your usage. Can you tell me:

  • Approximately how many .go files are in your workspace (this depends on which folder was resolved as your workspace folder).
  • Do you experience this memory usage immediately, or does it take a while to build up? There was a large memory leak that will be fixed in the next gopls release (v0.9.2), so I'm curious if you may be experiencing that.

Unfortunately, at the moment 5gb is not unheard-of memory usage when working on large projects, such as Kubernetes. We're working on this, but it is a long-term project.

@findleyr findleyr added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 5, 2022
@codisms
Copy link
Author

codisms commented Aug 5, 2022

Unfortunately, I don't have a great way to answer the first question. The number is hard to know because of the libraries being referenced, but the project itself has 3500 Go files, though not all referenced in all parts of the code. One thing I can say is that we have a single file generated by gqlgen that has 256k lines in it and is probably a big culprit.

Initially loading the project causes the machine to spin for a good five minutes or so and it'll spike to 3-4 GB pretty easily. It'll settle around 2-3 after a while but spikes as the files are edited. Multiple files could be changed at the same time with multiple vim windows going on. I do have the gopls server running for quite a long time though.

@findleyr
Copy link
Contributor

findleyr commented Aug 5, 2022

Thanks.

I expect you'll see it stay in the the ~3gb range after gopls@v0.9.2 is released. I'll take a look at the heap dump you've provided to see if I can understand this better.

@findleyr findleyr modified the milestones: Unreleased, gopls/later Aug 8, 2022
@findleyr findleyr removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 8, 2022
@findleyr findleyr modified the milestones: gopls/later, gopls/v0.10.0 Aug 8, 2022
@findleyr
Copy link
Contributor

findleyr commented Aug 8, 2022

Putting this in the next milestone for additional triage.

@galileo-pkm
Copy link

Not much has changed with this release. gopls remains a memory hog.
My "template" project without any real code and gopls eats up 2.2G.
With a few medium size project in VSCode workspace it regularly goes over 8GB.

gopls v0.9.4, go.mod for the project and the diagnostics file attached if anyone cares.
gopls.35536-2GiB-withnames.zip
go.mod.gz

@switchupcb
Copy link

switchupcb commented Nov 25, 2022 •
edited

Do you experience this memory usage immediately, or does it take a while to build up?
— @findleyr

I don't ever hit 1 GB of RAM (cause I'm not analyzing 256 KLOC). However, I noticed when running VSCode for a long time that gopls will freeze up and stop analyzing code. I kill the process (which shows higher memory consumption than usual) using Task Manager, which restarts gopls and it works for some time. Then, the cycle repeats.

Maybe a short term solution for the large projects is to have a mode that only analyzes specific files (i.e opened) or some way to exclude them (directory, whatever makes sense, etc).

Update: I hit 2GB on a 1500 line file.

@findleyr findleyr modified the milestones: gopls/later, gopls/v0.12.0 May 29, 2023
@findleyr
Copy link
Contributor

gopls@v0.12.0-pre.3 significantly reduces gopls' memory usage, up to an order of magnitude on larger repositories.

Closing this as resolved. Please open a new issue if you are still encountering memory issues with the v0.12.0 release (the final release is scheduled for this week).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Milestone
gopls/v0.12.0
Development

No branches or pull requests
6 participants
@codisms @galileo-pkm @gopherbot @findleyr @switchupcb @bkessler-go