-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Thread and Memory Leak in Kubelet and Go Program with Multiple Goroutines #68993
Comments
Yes, this is current expected behavior. Closing this issue as a dup of #14592. Thanks. |
Note that Go does not hold onto all memory forever (though it will keep the thread stacks since it doesn't exit the threads). Memory freed by the GC isn't immediately returned to the OS (so it can be reused quickly), but the runtime does release excessive memory to the OS eventually. The amount of memory to release is informed by GOGC and GOMEMLIMIT. If GOMEMLIMIT seems to have no effect, then it may be worth looking at the /gc and /memory metrics in cc @mknyszek |
Go version
go1.20.5 linux/amd64
Output of
go env
in your module/workspace:What did you do?
I’ve observed a persistent issue with thread and memory retention in both the Kubernetes Kubelet and a custom Go program that utilizes multiple goroutines. After workloads are terminated, the Go runtime appears to retain OS threads and memory, which aren’t released as expected, leading to elevated thread counts and memory usage over time.
Steps to Reproduce:
Custom Go Program:
I have experimented with adjusting the following Go runtime environment variables in the
kubelet
config file:GOMAXPROCS
: Set to different values to control the number of CPU cores.GOMEMLIMIT
: Set to a low value to force aggressive memory reclamation.GOGC
: Set to a low value (e.g., 50) to increase the frequency of garbage collection.However, these adjustments did not affected the thread and memory.
Is this behavior expected due to some inherent Go runtime behavior, or could this be an actual bug in the runtime’s thread and memory management? Any guidance on further debugging steps or potential workarounds would be greatly appreciated.
What did you see happen?
Kubelet: After workload termination, the Kubelet process retains a high number of OS threads and does not release the associated memory, resulting in persistent elevated resource usage.
Go Program: Similarly, in the custom Go program, OS threads and memory are not released after the goroutines have completed, leading to the same issue of resource retention.
What did you expect to see?
In both cases, I expected that once the workload (in Kubelet) or the goroutines (in the Go program) were completed, the Go runtime would release the idle OS threads and free up the allocated memory, returning the process to its baseline resource usage.
The text was updated successfully, but these errors were encountered: