x/tools/gopls: uses too much memory (60G) #37790

bobotu · 2020-03-11T07:55:35Z

What version of Go are you using (`go version`)?

$ go version
go version go1.14 linux/amd64

Does this issue reproduce with the latest release?

Yes

golang.org/x/tools/gopls v0.3.3
    golang.org/x/tools/gopls@v0.3.3 h1:mTFqRDJQmpSsgDDWvbtGnSva1z9uX2XcDszSWa6DhBQ=

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/tuzi/.cache/go-build"
GOENV="/home/tuzi/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/tuzi/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib64/go/1.14"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib64/go/1.14/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/tuzi/projects/tidb/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build266011737=/tmp/go-build -gno-record-gcc-switches"

VS Code settings

    "go.useLanguageServer": true,
    "go.autocompleteUnimportedPackages": true,
    "go.formatTool": "goimports",
    "go.lintTool": "golangci-lint",
    "[go]": {
        "editor.formatOnSave": true,
        "editor.codeActionsOnSave": {
            "source.organizeImports": false,
        },
        // Optional: Disable snippets, as they conflict with completion ranking.
        "editor.snippetSuggestions": "none",
    },
    "[go.mod]": {
        "editor.formatOnSave": true,
        "editor.codeActionsOnSave": {
            "source.organizeImports": true,
        },
    },
    "go.languageServerExperimentalFeatures": {
        "format": false,
        "diagnostics": true,
        "documentLink": true
    },
    "go.languageServerFlags": [
        "-rpc.trace",
        "serve",
        "--debug=localhost:6060"
    ],
    "gopls": {
        // Add parameter placeholders when completing a function.
        "usePlaceholders": false,
        // If true, enable additional analyses with staticcheck.
        // Warning: This will significantly increase memory usage.
        "staticcheck": false,
    },

What did you do?

editor: VS Code
go project: pingcap/tidb
After about half an hour, the gopls eat all of my RAM.

What did you expect to see?

Not use too many memory

What did you see instead?

atop says it has used nearly all RAM on my machine.

heap profile

The text was updated successfully, but these errors were encountered:

gopherbot · 2020-03-11T07:55:43Z

Thank you for filing a gopls issue! Please take a look at the Troubleshooting guide, and make sure that you have provided all of the relevant information here.

heschi · 2020-03-11T16:17:49Z

Thanks for the report. There should be some files named gopls.PID- in your temporary directory, named after heap sizes. Please upload the pair for the largest heap -- I would like to see a goroutine dump that matches this heap dump.

bobotu · 2020-03-11T16:28:48Z

@heschik gopls.14043-53GiB-heap.pb.gz and gopls.14043-53GiB-goroutines.txt

heschi · 2020-03-11T16:38:54Z

Thanks again.

I don't see any signs of a bug here but this is a massive heap. Are you using gopls in a large monorepo, or do you have huge code-generated files?

bobotu · 2020-03-11T16:48:00Z

I'm working with TiDB codebase.
it's a huge project, some dependency may contains huge protobuf generated code.

paulsubrata55 · 2020-06-05T04:38:56Z

Same here. I had to uninstall the gopls as my system went for a toss. I had to hard reboot my system.
I also have huge protobuf generated code.

heschi · 2020-06-05T18:15:59Z

@paulsubrata55 I'm sorry you're having trouble, but without a profile or a repro there is absolutely nothing anyone can do to help you. Please follow the instructions above.

heschi · 2020-06-11T23:16:07Z

@bobotu I believe some changes I landed today have improved memory usage on large projects substantially. Could you try master using the instructions at https://github.com/golang/tools/blob/master/gopls/doc/user.md#unstable-versions?

xhebox · 2020-06-28T13:49:54Z

@heschik opening TiDB at the root directory will eat up all my 8 GB memory(finally killed by OOM). It is a little better if you just use gopls in a very small directory(< 1 GB memory footprint).

I think gopls needs to be more "lazy" about the imported package(at least provide an option to avoid pulling all dependencies)...

heschi · 2020-06-29T23:17:22Z

@xhebox I'm sorry you're having trouble, but I cannot reproduce the problem. Please follow the instructions above to collect memory usage information.

xhebox · 2020-06-30T02:21:00Z

@heschik This is the memory profile: at github.com/pingcap/tidb/kv directory, open memdb_buffer.go file, with gopls@v0.0.0-20200626171337-aa94e735be7f.
Since open something at root will crash my computer, here is one small profile, 300-400M inuse, 600-700M alloc.

mem.txt

I don't see any signs of a bug here but this is a massive heap.

Yes, you are right that maybe there is no bug. But I believe that gopls may have parsed all files under kv directory.

Because memdb_buffer.go only imports tidb/kv/memdb(which costs about 100-150MB alloc), three standard libraries, and pingcap/errors. I expect it is about 200MB alloc, but it turns to be 600-700MB alloc.

I would like to let gopls only parse this opening file, lazily load type info(e.g. gopls only loads the info of Iterator when I am typing Ite), or even remove loaded type info aggressively. Or stop parsing new files when a user specified threshold is exceeded.

ldeng-ustc · 2020-07-06T12:53:21Z

I would like to let gopls only parse this opening file, lazily load type info(e.g. gopls only loads the info of Iterator when I am typing Ite), or even remove loaded type info aggressively. Or stop parsing new files when a user specified threshold is exceeded.

I couldn't agree more. No one needs info of all packages all the time. Instead, memory resource are more valuable.

ericwq · 2020-07-06T14:10:38Z

this is my go.mod file, after a few gd jump, it consume 4-5G memory, it hurt my build speed badly :

qiwang@Einstein droplocal % more go.mod
module github.com/ericwq/droplocal

go 1.14

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/goftp/file-driver v0.0.0-20180502053751-5d604a0fc0c9
        github.com/goftp/server v0.0.0-20190712054601-1149070ae46b
        github.com/grandcat/zeroconf v1.0.0
        github.com/jlaffaye/ftp v0.0.0-20200602180915-5563613968bf
        github.com/kr/pretty v0.1.0 // indirect
        golang.org/x/net v0.0.0-20200226121028-0de0cce0169b // indirect
        golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e // indirect
        gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect
        gopkg.in/yaml.v2 v2.2.4 // indirect
)

heschi · 2020-07-06T16:29:16Z

The original issue reporter is no longer responding and I believe this issue is resolved or at least improved, so I'm closing the issue.

gopls is currently expected to use ~1GiB on a project of any significant size. Memory usage below that is unfortunately not a bug. We need to load a lot of information about each package to type check the project and to support features like Find References.

As I've said multiple times on this issue, I need the gopls.* files from /tmp to investigate memory problems further. Please file a new issue with one attached if you're having trouble, otherwise there is nothing anyone can do to help you.

gopherbot added this to the Unreleased milestone Mar 11, 2020

gopherbot added Tools This label describes issues relating to any tools in the x/tools repository. gopls Issues related to the Go language server, gopls. labels Mar 11, 2020

ALTree changed the title ~~x/tools/gopls: use too many memory (60G)~~ x/tools/gopls: uses too much memory (60G) Mar 11, 2020

ALTree added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 11, 2020

stamblerre modified the milestones: Unreleased, gopls/v0.4.0 Mar 11, 2020

stamblerre modified the milestones: gopls/v0.4.0, gopls/v0.5.0 Apr 2, 2020

heschi added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 16, 2020

hyangah mentioned this issue Jul 5, 2020

suggest is too slow golang/vscode-go#281

Closed

heschi closed this as completed Jul 6, 2020

stamblerre modified the milestones: gopls/v0.5.0, gopls/v0.4.4 Jul 9, 2020

golang locked and limited conversation to collaborators Jul 9, 2021

gopherbot added the FrozenDueToAge label Jul 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/gopls: uses too much memory (60G) #37790

x/tools/gopls: uses too much memory (60G) #37790

bobotu commented Mar 11, 2020

gopherbot commented Mar 11, 2020

heschi commented Mar 11, 2020

bobotu commented Mar 11, 2020

heschi commented Mar 11, 2020

bobotu commented Mar 11, 2020 •

edited

paulsubrata55 commented Jun 5, 2020

heschi commented Jun 5, 2020

heschi commented Jun 11, 2020

xhebox commented Jun 28, 2020 •

edited

heschi commented Jun 29, 2020

xhebox commented Jun 30, 2020 •

edited

ldeng-ustc commented Jul 6, 2020

ericwq commented Jul 6, 2020

heschi commented Jul 6, 2020

x/tools/gopls: uses too much memory (60G) #37790

x/tools/gopls: uses too much memory (60G) #37790

Comments

bobotu commented Mar 11, 2020

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

gopherbot commented Mar 11, 2020

heschi commented Mar 11, 2020

bobotu commented Mar 11, 2020

heschi commented Mar 11, 2020

bobotu commented Mar 11, 2020 • edited

paulsubrata55 commented Jun 5, 2020

heschi commented Jun 5, 2020

heschi commented Jun 11, 2020

xhebox commented Jun 28, 2020 • edited

heschi commented Jun 29, 2020

xhebox commented Jun 30, 2020 • edited

ldeng-ustc commented Jul 6, 2020

ericwq commented Jul 6, 2020

heschi commented Jul 6, 2020

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

bobotu commented Mar 11, 2020 •

edited

xhebox commented Jun 28, 2020 •

edited

xhebox commented Jun 30, 2020 •

edited