Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thread and Memory Leak in Kubelet and Go Program with Multiple Goroutines #68993

Closed
pranav-pandey0804 opened this issue Aug 21, 2024 · 3 comments

Comments

@pranav-pandey0804
Copy link

pranav-pandey0804 commented Aug 21, 2024

Go version

go1.20.5 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/nec/.cache/go-build"
GOENV="/home/nec/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/nec/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/nec/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.5"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3908633320=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I’ve observed a persistent issue with thread and memory retention in both the Kubernetes Kubelet and a custom Go program that utilizes multiple goroutines. After workloads are terminated, the Go runtime appears to retain OS threads and memory, which aren’t released as expected, leading to elevated thread counts and memory usage over time.

Steps to Reproduce:

  1. Kubelet:
  • Deploy a Kubernetes cluster with Kubelet running as a pod.
  • Apply a workload that creates high CPU and memory pressure (e.g., a deployment with containers running frequent readiness probes).
  • Monitor the number of threads and memory usage of the Kubelet process before, during, and after the workload.
  • Observe that threads and memory are not released even after the workload is terminated.

Custom Go Program:

  • Write a Go program that spawns a large number of goroutines, each performing some work (e.g., simple computations or I/O tasks).
  • Monitor the thread count and memory usage after the goroutines complete.
  • Observe that the OS threads and memory usage remain elevated.

I have experimented with adjusting the following Go runtime environment variables in the kubelet config file:

  • GOMAXPROCS: Set to different values to control the number of CPU cores.
  • GOMEMLIMIT: Set to a low value to force aggressive memory reclamation.
  • GOGC: Set to a low value (e.g., 50) to increase the frequency of garbage collection.

However, these adjustments did not affected the thread and memory.

Is this behavior expected due to some inherent Go runtime behavior, or could this be an actual bug in the runtime’s thread and memory management? Any guidance on further debugging steps or potential workarounds would be greatly appreciated.

What did you see happen?

Kubelet: After workload termination, the Kubelet process retains a high number of OS threads and does not release the associated memory, resulting in persistent elevated resource usage.

Go Program: Similarly, in the custom Go program, OS threads and memory are not released after the goroutines have completed, leading to the same issue of resource retention.

What did you expect to see?

In both cases, I expected that once the workload (in Kubelet) or the goroutines (in the Go program) were completed, the Go runtime would release the idle OS threads and free up the allocated memory, returning the process to its baseline resource usage.

@ianlancetaylor
Copy link
Member

Yes, this is current expected behavior. Closing this issue as a dup of #14592. Thanks.

@ianlancetaylor ianlancetaylor closed this as not planned Won't fix, can't repro, duplicate, stale Aug 21, 2024
@prattmic
Copy link
Member

Note that Go does not hold onto all memory forever (though it will keep the thread stacks since it doesn't exit the threads).

Memory freed by the GC isn't immediately returned to the OS (so it can be reused quickly), but the runtime does release excessive memory to the OS eventually. The amount of memory to release is informed by GOGC and GOMEMLIMIT. If GOMEMLIMIT seems to have no effect, then it may be worth looking at the /gc and /memory metrics in runtime/metrics to see where the memory is going.

cc @mknyszek

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants