Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools/gopls: significant memory usage in kubernetes/vendor/k8s.io/client-go #40651

Closed
microyahoo opened this issue Aug 8, 2020 · 9 comments
Closed

x/tools/gopls: significant memory usage in kubernetes/vendor/k8s.io/client-go #40651

microyahoo opened this issue Aug 8, 2020 · 9 comments
Labels
FrozenDueToAge gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
gopls/v0.5.0

Comments

@microyahoo
Copy link

microyahoo commented Aug 8, 2020 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.14.4 linux/amd64

(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ gopls version
golang.org/x/tools/gopls 0.4.4
    golang.org/x/tools/gopls@v0.4.4 h1:8djGYsaZ0ByP0vaXg4T+mnyfDcHpWKSZ+tpQSGv9ahk=

Does this issue reproduce with the latest release?

it can be reproduced in my VMWare centos.

(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ uname -r
3.10.0-1127.10.1.el7.x86_64
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/admin/.cache/go-build"
GOENV="/home/admin/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/admin/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/admin/go/src/k8s.io/kubernetes/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build351835036=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I just opend the client-go directory, the memory usage will be increased dramatically, after a while, it will be fixed at a certain value, in my os, about 6G. I'm not sure whether the symbolic link affect the gopls, because in k8s vendor directory, it's a link to staging directory.

(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        454M        6.1G         12M        1.0G        6.9G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ps -ef | grep gopls
admin     13612   8366  0 08:38 pts/2    00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox gopls
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        455M        6.1G         12M        1.0G        6.9G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ vim vendor/k8s.io/client-go

[1]  + 13685 suspended  nvim vendor/k8s.io/client-go
(ENV_3.7.6) ➜ /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        901M        5.7G         12M        1.0G        6.5G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        780M        5.8G         12M        1.0G        6.6G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        1.6G        5.0G         12M        1.0G        5.7G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ fg
[1]    13685 continued  nvim vendor/k8s.io/client-go

[1]  + 13685 suspended  nvim vendor/k8s.io/client-go
(ENV_3.7.6) ➜ /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        3.9G        2.6G         12M        1.0G        3.4G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ fg
[1]    13685 continued  nvim vendor/k8s.io/client-go

[1]  + 13685 suspended  nvim vendor/k8s.io/client-go
(ENV_3.7.6) ➜ /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        3.9G        2.6G         12M        1.0G        3.4G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ fg
[1]    13685 continued  nvim vendor/k8s.io/client-go

[1]  + 13685 suspended  nvim vendor/k8s.io/client-go
(ENV_3.7.6) ➜ /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        4.7G        1.9G         12M        1.0G        2.6G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        5.4G        1.2G         12M        1.0G        1.9G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        5.4G        1.2G         12M        1.0G        1.9G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ fg
[1]    13685 continued  nvim vendor/k8s.io/client-go

[1]  + 13685 suspended  nvim vendor/k8s.io/client-go
(ENV_3.7.6) ➜ /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        6.1G        520M         12M        1.0G        1.3G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        6.1G        520M         12M        1.0G        1.3G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ ps -ef | grep gopls
admin     13707  13685  0 08:39 ?        00:00:00 /home/admin/go/bin/gopls -remote=auto
admin     13721  13688 28 08:39 ?        00:01:13 gopls
admin     13740  13707  2 08:39 ?        00:00:06 /home/admin/go/bin/gopls serve -listen unix;/tmp/gopls-dcea82-daemon.admin -listen.timeout 1m0s
admin     16846   8366  0 08:43 pts/2    00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox gopls

(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ ll vendor/k8s.io/client-go
lrwxrwxrwx. 1 admin 34 Aug  8 08:32 vendor/k8s.io/client-go -> ../../staging/src/k8s.io/client-go/

The memory usage comes to normal after stop gopls

(ENV_3.7.6) ➜ /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        1.2G        5.4G         12M        1.0G        6.2G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ ps -ef | grep gopls
admin     13740      1  0 08:39 ?        00:00:22 /home/admin/go/bin/gopls serve -listen unix;/tmp/gopls-dcea82-daemon.admin -listen.timeout 1m0s
admin     20043   8366  0 09:27 pts/2    00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox gopls
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        456M        6.1G         12M        1.0G        6.9G
Swap:          7.9G          0B        7.9G

What did you expect to see?

Keep memory utilization at a low level when using gopls.

What did you see instead?

the memory usage increase dramtically.
@gopherbot gopherbot added the gopls Issues related to the Go language server, gopls. label Aug 8, 2020
@microyahoo microyahoo changed the title gopls: after opening the kubernetes/rvendor/k8s.io/client-go for a while, the memory usage will increase dramatically gopls: after opening the kubernetes/vendor/k8s.io/client-go for a while, the memory usage will increase dramatically Aug 8, 2020
@stamblerre stamblerre added this to the gopls/v.0.4.5 milestone Aug 8, 2020
@stamblerre stamblerre changed the title gopls: after opening the kubernetes/vendor/k8s.io/client-go for a while, the memory usage will increase dramatically x/tools/gopls: significant memory usage in kubernetes/vendor/k8s.io/client-go Aug 8, 2020
@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Aug 8, 2020
@stamblerre
Copy link
Contributor

stamblerre commented Aug 8, 2020 •
edited

Thanks for the report, @microyahoo. @heschik has been working on improving gopls's memory usage (https://golang.org/cl/242838), so we will try out your repro case once that change is merged.

@heschi
Copy link
Contributor

heschi commented Aug 11, 2020

The situation looks good to me at master. You can try for yourself with the instructions on https://github.com/golang/tools/blob/master/gopls/doc/user.md#unstable-versions, or wait for the next release.

@stamblerre stamblerre added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 11, 2020
@fahadbr
Copy link

fahadbr commented Aug 12, 2020

can confirm that using master solves a lot of memory growth issues 👍

@microyahoo
Copy link
Author

microyahoo commented Aug 13, 2020 •
edited

@heschik Thanks for your quick response, I have try it, seems it still keeps large memory usage as below.

(ENV_3.7.6) ➜ /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        4.1G        2.8G         12M        730M        3.2G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        6.3G        505M         12M        845M        1.0G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ gopls version
golang.org/x/tools/gopls master
    golang.org/x/tools/gopls@v0.0.0-20200812231640-9176cd30088c h1:/vI4IGFwuBZgxpQLSzEinIShNNDySWUB3i8Q3bXtHNk=
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        6.6G        217M         12M        845M        773M
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        6.6G        218M         12M        845M        773M
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ jobs
[1]  + suspended  nvim vendor/k8s.io/client-go
(ENV_3.7.6) 🍺 /home/admin/go/src/k8s.io/kubernetes ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        6.6G        217M         12M        845M        773M
Swap:          7.9G          0B        7.9G

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  3664 admin     20   0 4508516   3.2g   9020 S   4.7 41.8   2:40.55 gopls
  3671 admin     20   0 4180456   3.0g   8760 S   4.3 39.0   2:26.01 gopls
  3630 admin     20   0  754800  37896  16148 S   0.0  0.5   0:01.08 node
   889 root      20   0  358764  29556   7072 S   0.0  0.4   0:00.49 firewalld
  1252 root      20   0 1010824  20764  11460 S   0.0  0.3   0:00.31 libvirtd
  3641 admin     20   0  906992  18604   5848 S   0.0  0.2   0:00.23 gopls
  1248 root      20   0  574308  17508   6152 S   0.0  0.2   0:00.71 tuned
   827 polkitd   20   0  616624  15660   4948 S   0.0  0.2   0:00.15 polkitd
  3627 admin     20   0  210740  13828   3756 T   0.0  0.2   0:00.55 nvim

I'm confused if I open the other repo like etcd, the memory usage is stable in 2.2G, but if I open kubernetes/vendor/k8s.io/client-go, the memory will be run out quickly.

(ENV_3.7.6) ➜ /home/admin/go/src/github.com/etcd-io/etcd ☞ git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        2.2G        4.4G         12M        992M        5.1G
Swap:          7.9G          0B        7.9G
(ENV_3.7.6) 🍺 /home/admin/go/src/github.com/etcd-io/etcd ☞ git:(master) ✗ jobs
[1]  + suspended  nvim
(ENV_3.7.6) 🍺 /home/admin/go/src/github.com/etcd-io/etcd ☞ git:(master) ✗ ps -ef | grep gopls
admin      5204   5190  0 11:44 ?        00:00:00 /home/admin/go/bin/gopls -remote=auto
admin      5229   5204  4 11:44 ?        00:00:25 /home/admin/go/bin/gopls serve -listen unix;/tmp/gopls-be61fc-daemon.admin -listen.timeout 1m0s
admin      5230   5193  2 11:44 ?        00:00:16 gopls
admin      5893   2347  0 11:54 pts/0    00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox gopls
(ENV_3.7.6) 🍺 /home/admin/go/src/github.com/etcd-io/etcd ☞ git:(master) ✗ gopls version
golang.org/x/tools/gopls master
    golang.org/x/tools/gopls@v0.0.0-20200812231640-9176cd30088c h1:/vI4IGFwuBZgxpQLSzEinIShNNDySWUB3i8Q3bXtHNk=

@heschi
Copy link
Contributor

heschi commented Aug 13, 2020

None of your commands actually show the memory usage of gopls, so I'm not sure what I'm supposed to be looking at here.

When gopls uses a lot of memory, it writes debugging information to /tmp/gopls.*-withnames.zip. Please attach the one corresponding to the highest heap size here.

@microyahoo
Copy link
Author

Hi @heschik , I'm really sorry for my late reply. I have attached the log, please do not hesitate to contact me in case of any query.
gopls.log.zip

@heschi
Copy link
Contributor

heschi commented Aug 18, 2020

You said that you opened the client-go directory, but the zip file that you uploaded looks more like it's dealing with all of Kubernetes. Can you confirm which directory you're opening?

Kubernetes is a huge project and unfortunately it's going to use a lot of memory. https://golang.org/cl/248180 will help a tiny bit if you download the very latest master, but right now I don't see any big surprises. We will continue to pay attention to memory usage but I don't think things are going to get much better right now.

@microyahoo
Copy link
Author

Thanks for your analysis. The client-go is underneath the Kubernetes project.
https://github.com/kubernetes/kubernetes/blob/master/vendor/k8s.io/client-go

@heschi
Copy link
Contributor

heschi commented Aug 18, 2020

The directory you open as the workspace directory in VS Code defines the project that gopls will load and analyze. You appear to be opening the top-level Kubernetes directory as your workspace; if you open the client-go directory instead, gopls will use much less memory.

If you have any further questions please update to the latest master, and include both a fresh memory debug zip and the gopls server logs, as described in https://github.com/golang/tools/blob/master/gopls/doc/troubleshooting.md#capturing-logs.

@golang golang locked and limited conversation to collaborators Sep 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
gopls/v0.5.0
Development

No branches or pull requests
5 participants
@stamblerre @gopherbot @microyahoo @fahadbr @heschi