gopls doesn't invoke the Staticcheck binary. Even with Staticcheck enabled in gopls, the checks would execute directly in the gopls process. So this doesn't seem like a gopls issue.

This may be a vscode-go issue, however.

CC @hyangah

Agree that it's likely the vscode-go issue rather than gopls or go - @bschaatsbergen can you verify staticchecks are the children of a VSCode process?
One workaround is to disable staticcheck invocation by vscode-go, but delegate to gopls by enabling
"gopls": { "ui.diagnostics.staticcheck": true }

I'm seeing similar here. downgrading to 0.8.0 seems to work with setting "go.toolsManagement.autoUpdate": false, in vscode settings

Hi, fwiw I don't think gopls@v0.8.0 should make a difference relative to gopls@v0.8.1. @ElanHasson are you sure this matters?

Several internal folks here have reported gopls as being slow for them and eating memory. As you mentioned this may be a vscode-go issue.

Downgrading seems to have resolved the issue for us. I did a quick search in vscode issues list and couldn't find anything relevant.

What is the best way to gather data on this to help diagnose the issue (or rule out gopls)?

Gopls does use a lot of memory, but I'd be a bit surprised if it had a regression in v0.8.1, as that was mostly a bugfix release.

If you could run gopls@v0.8.0 on your codebase and report its RSS once it has finished loading, then compare with gopls@v0.8.1, that would be very helpful.

Which version of Go are you using, and do you use go workspace? go.work?

❯ ps -o pid,vsz,rss,%mem,command -p $(ps -m -o pid=28227 | head -n1)
  PID    VSZ   RSS %MEM COMMAND
28227 3781784 1705460 10.5 /home/elan/go/bin/gopls -mode=stdio
❯ go version
go version go1.18 linux/amd64


❯ ps -o pid,vsz,rss,%mem,command -p $(ps -m -o pid=23130 | head -n1)
  PID    VSZ   RSS %MEM COMMAND
23130 14398852 12438852 76.5 /home/elan/go/bin/gopls -mode=stdio
❯ gopls version
golang.org/x/tools/gopls v0.8.1
    golang.org/x/tools/gopls@v0.8.1 h1:q5nDpRopYrnF4DN/1o8ZQ7Oar4Yd4I5OtGMx5RyV2/8=

  PID    VSZ   RSS %MEM COMMAND
13322 13516860 10040104 61.8 /home/elan/go/bin/gopls -mode=stdio
❯ gopls version
golang.org/x/tools/gopls v0.8.2
    golang.org/x/tools/gopls@v0.8.2 h1:+b7X2FDwmP7vxyYNOHe/ZFkecN5X8OtAgkNtuCr86wk=

I'd say this should be closed as the sluggishness has gone away.

Thanks for following up!

so, hate to be "that guy"...

Output from VSCode:

Starting linting the current package at /home/elan/repos/cthulhu/docode/src/do/teams/insights/executioner/pkg/execute
/home/elan/repos/cthulhu/docode/src/do/teams/insights/executioner/pkg/execute>Finished running tool: /home/elan/go/bin/staticcheck

Failed while handling 'onDidChangeTextDocument': TypeError: Cannot read properties of undefined (reading 'children')

@ElanHasson your screenshot shows a lot of gopls processesthreads (using a ton of memory), but your output relates to staticcheck.

Can you clarify what you're pointing out? Do you know why you have so many gopls processesthreads?

EDIT: threads, not processes

Hi @findleyr, I'm not sure-- I imagine it's a "feature" of VSCode.

I'm not sure if this is a VSCode problem (the staticcheck or multiple gopls). I figured I'd add details here for discoverability.

The previous RSS numbers I dropped were very low compared to these above. I'm unfamiliar with the internals of gopls or staticcheck nor how VSCode uses them.

I'm going to log an issue in https://github.com/microsoft/vscode and link to this for reference and operate under the impression it's a VSCode issue first.

I'd still say leave this closed unless the VSCode folks come back and say something otherwise.

I don't think this has anything to do with microsoft/vscode. It's either gopls or vscode-go (both of which are us :) )

Reopening to continue investigation:

• can you please check which process started all those gopls instances?
• can you try killing them? Are they all restarted?
• how many vscode workspaces do you have open?
• which directory did you open as your workspace? gopls can use a lot of memory if opened to e.g. your home directory

Thanks for clarifying ownership, much appreciated.

Versions:

❯ gopls version
golang.org/x/tools/gopls v0.8.3
    golang.org/x/tools/gopls@v0.8.3 h1:Mxm94ix8oSARQ6svioO6SxKEYWT/VCP54/448LOHzrk=

❯ /home/elan/go/bin/staticcheck  -version
staticcheck 2022.1 (v0.3.0)

❯ go version
go version go1.18 linux/amd64

[golang.Go ](https://marketplace.visualstudio.com/items?itemName=golang.Go) = v0.32.0

Visual Studio Code:

Version: 1.66.1 (user setup)
Commit: 8dfae7a5cd50421d10cd99cb873990460525a898
Date: 2022-04-06T14:50:12.141Z (6 days ago)
Electron: 17.2.0
Chromium: 98.0.4758.109
Node.js: 16.13.0
V8: 9.8.177.11-electron.0
OS: Windows_NT x64 10.0.22000

I'm using VSCode WSL-Remote on Windows 11.

1. Seems to be vscode and then gopls started the rest:

2. killall -9 gopls kills them all and they are restarted, memory slowly rising over the course of a few minutes
3. I had multiple open VSCodes open, closed them and now I only have one workspace with multiple folders/repos (some folders in same monorepo)
4. My .code-workspace is in ~/repos/workspaces and my others are in ~/repos ( ../ = ~/repos):

{
	"folders": [
		{
			"path": "../cthulhu/docode/src/do/teams/paas/apps"
		},
		{
			"path": "../godo"
		},
		{
			"path": "../product-docs"
		},
		{
			"path": "../cthulhu/docode/src/do/teams/serverless"
		},
		{
			"path": "../openapi"
		},
		{
			"path": "../workspaces"
		},
		{
			"path": "../cthulhu/docode/src/do/teams/insights"
		},
		{
			"path": "../edge-loadbalancers"
		},
		{
			"path": "../dographql"
		}
	],

Can you clarify what you're pointing out? Do you know why you have so many gopls processes?

These are threads, not processes. And the memory statistics for threads mirror that of the owning process. So that's a single gopls process, with several threads, and the gopls process uses 9.6 GiB in total.

Thanks for pointing that out.

After some time it becomes impossible to save any files and it hangs during the static check (at least it seems to).

When that happens I can't use any of the go behavior within vscode. For example, finding references, and I get red squiggly lines on valid code that I just added.

These are threads, not processes.

🤦, thanks, I should have noticed the RSS was all the same...

I'm afraid 9.6 GiB is not unheard of for large workspaces. For reference gopls uses 3-6gb for kubernetes, depending on whether staticcheck is enabled. I recommend reducing the size of your workspace to modules that you are actively working on, if possible.

its worth noting the code in my workspace isn't all go code and it is definitely less code than k8s.

does gopls have the ability to dump any runtime info?

its worth noting the code in my workspace isn't all go code and it is definitely less code than k8s.

There may well be a bug related to that. Can you share a sense of the size of the workspace, in terms of:

• number of total files in workspace folders
• number of go files in the workspace folders

does gopls have the ability to dump any runtime info?

Gopls automatically dumps heap to files of the form /tmp/gopls.<PID>-NGiB-(with|no)names.zip, which you can use to investigate. (upon unzipping, you can look at the heap profile with go tool pprof and open the html file, which contains estimates of relative memory usage by type-checked package. In the latter, the units are meaningless but it can be useful to see whether there are packages in there that should not be there.

thanks @findleyr will capture data when i hit the issue again.

It's going to be difficult to know what should be loaded and not due to that monorepo life.

aaand i hit it again :D

Sometimes I'm able to run the test despite the red squiggles, other times not. Killing gopls and retrying allows me to run again:

File Counts

/home/elan/repos/cthulhu/docode/src/do/teams/paas/apps
+ find . -type f
+ wc -l
1402
+ find . -type f -name '*.go'
+ wc -l
971

/home/elan/repos/cthulhu/docode/src/do/teams/serverless
+ find . -type f
+ wc -l
497
+ find . -type f -name '*.go'
+ wc -l
264

/home/elan/repos/godo
+ wc -l
+ find . -type f
202
+ wc -l
+ find . -type f -name '*.go'
82

/home/elan/repos/product-docs
+ find . -type f
+ wc -l
8634
+ find . -type f -name '*.go'
+ wc -l
0

/home/elan/repos/edge-loadbalancers
+ find . -type f
+ wc -l
260
+ find . -type f -name '*.go'
+ wc -l
0

/home/elan/repos/dographql
+ find . -type f
+ wc -l
1408
+ find . -type f -name '*.go'
+ wc -l
0

/home/elan/repos/cthulhu/docode/src/do/teams/insights
+ find . -type f
+ wc -l
1416
+ find . -type f -name '*.go'
+ wc -l
825

/home/elan/repos/dographql
+ find . -type f
+ wc -l
1408
+ find . -type f -name '*.go'
+ wc -l
0

/home/elan/repos/openapi
+ find . -type f
+ wc -l
1379
+ find . -type f -name '*.go'
+ wc -l
0

memory usage aside.. I can run the test and the IDE isn't getting updated static checking (?) and the red squiggles still appear.

I'm not really sure how to determine what should and should not be there.. there are loads of transitive dependencies.

I've never used go tool pprof before, but running web results in powershell being executed? I'm running from Windows Terminal in Ubuntu, not in a WSL windows mount... so I think it thinks i'm on windows?

Interesting... I killed gopls and vscode didn't like it and dumped its stack:

Why are those repos (living in ~/repos) even being noticed by gopls, this may be the reason why it's using so much memory, it's loading lots of unrelated folders?

/home/elan/repos
+ find . -type f
+ wc -l
264795
+ find . -type f -name '*.go'
+ wc -l
46962

Update: Had a go.mod in ~/repos, not sure if that matters

I'm at DigitalOcean and that this is effectively a continuation of the work @bhcleek started with the team on
#45363.

Thank you for the data, and sorry for the slow turnaround.

Are you able to:

1. Upgrade to Go 1.18
2. Use a go.work that activates only the repos you care about? I.e. go work init <list of directories you want to use> from ~/repos, and open the ~/repos directory?

Can you share your gopls settings? Do you have experimentalWorkspaceModule: true?

Thanks @findleyr, no worries about the turnaround, between holidays, family, and life in general we all get busy 😃

We're on Go 1.18.1.

I'll investigate the the go.work stuff, it is new to me.

I personally do not have experimentalWorkspaceModule set.

Just checking in -- did using go.work help?

We're working on redesigning the core of gopls to finally address some of these scaling problems, but in the meantime go.work can help limit the scope of your workspace.

Hi,

Someone on the team is experimenting with go.work, I'll check in with them and respond here.

For me specifically, I increased the CPU to 16 cores and the problem went away. I watch the CPU spike when I use the show references functionality, sometimes for 10 seconds across all cores. Gopls taking up 50-60% of the 32GB ram.

This is using go 1.18.2 and latest versions of gopls and friends.

I've had this issue too, and in the end vscode-go became completely unusable for me. Now I've switched to GoLand and have experienced no performance issues at all.

@barrettj12 I'm sorry to hear that vscode-go became unusable for you. Unfortunately, there are aspects of gopls that scale with the size of the workspace, and so it will (currently) be unusable on large codebases.

However, we're working hard on performance improvements, and have landed several for the v0.9.0 release. Right now, these performance improvements are focused on CPU usages while typing, as well as optimizing workspaces that are narrowed using a go.work file. @ElanHasson I'd be very curious to hear if these help (go install golang.org/x/tools/gopls@v0.9.0-pre.2).

These optimizations still do not tackle the problem of type-checking the entire workspace. That will take longer to solve, though the "degradeClosed" memoryMode may help.

@findleyr, just installed, will monitor.

I will say after working for ~a month with the increased RAM and CPUs has been radically different.

I just installed the v0.9.0-pre.2 and I can say after a little usage, EVERYTHING is much snappier! The hover-over types/functions responds faster than I've ever seen, so does jumping to references.

Memory Usage is:
VIRT: 18G
RES: 15.6G
Relatively low CPU usage once everything is loaded.

Will update accordingly if anything changes.

@findleyr I can report back that v9.0-pre.2 seems to have fixed the issue. Every so often (maybe a few days) I have to restart VSCode because the language service (maybe ?) stops working.

@ElanHasson that's great news, thanks for reporting back.

We've got more improvements in the pipeline: nothing yet that will make an order-of-magnitude improvement, but several things that will reduce a significant fraction of load and memory.

I'm going to close this issue now since it's not specifically actionable, though of course performance remains a significant concern for gopls.

@findleyr thank you for your work on this, it is appreciated!