Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak in using slice of channels #43433

Closed
oghaffarinia opened this issue Dec 30, 2020 · 6 comments
Closed

Memory leak in using slice of channels #43433

oghaffarinia opened this issue Dec 30, 2020 · 6 comments
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@oghaffarinia
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.15.6 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/omid/.cache/go-build"
GOENV="/home/omid/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/omid/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/omid/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.15"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.15/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build972503043=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Creating many channels inside slice (and not using them), garbage collector does not free them
This is the code I run: https://play.golang.org/p/0OaEXpZukQ9

What did you expect to see?

I expect to see memory usage be dropped when not using created objects

What did you see instead?

I see that memory usage never goes down and more memory is reserved for each iteration (i.e. there is memory leak)

@ALTree
Copy link
Member

ALTree commented Dec 30, 2020

Please explain how exactly are you determining the amount of memory effectively used by the process, and thus that there's a leak.

Keep in mind that Go 1.15 on linux uses MADV_FREE to return memory, so the usual top/htop statistics will not be updated unless the OS actually reclaims the memory. But the Go program is not actually reserving the amount of memory shown. See #42330.

@ALTree ALTree added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Dec 30, 2020
@oghaffarinia
Copy link
Author

oghaffarinia commented Dec 30, 2020

This is how I determine amount of memory usage: (This code is run)

$ go run test-leak.go
Enter length (quit to exit): 
$ ps --pid=14499 -o size
 SIZE
102652
Enter length (quit to exit): 100000
Enter length (quit to exit): 
$ ps --pid=14499 -o size
 SIZE
10827872
Enter length (quit to exit): 100000
Enter length (quit to exit): 100000
Enter length (quit to exit): 
$ ps --pid=14499 -o size
 SIZE
17403580

And continuing this way, the memory usage keeps going up until the whole RAM is used and process is killed by kernel

$ go run test-leak.go
Enter length (quit to exit): 100000
Enter length (quit to exit): 100000
Enter length (quit to exit): 100000
signal: killed

I have 16GB RAM + 8GB swap which was fully consumed by entering 100000 for 3 times

@seankhliao
Copy link
Member

If you add explicit runtime.GC() calls it should work closer as to what you expect, only keeping ~10G of memory. leave it long enough and it clears out everything, which indicates it's not a leak, just GC not happening yet.

@hawell
Copy link

hawell commented Dec 30, 2020

you're allocating too much memory too fast. the scavenger wont return unused memory back to os until after a certain amount of time. (this used to be 5 minutes)
if you wait long enough (2 minutes for forced GC at most) you will that the memory is freed.
you can check this by using GODEBUG=gctrace=1
one more thing: this has nothing to do with channels
as @seankhliao mentioned you can use runtime.GC() (or debug.FreeOSMemory() ) explicitly

@randall77
Copy link
Contributor

Channels are 96 bytes, plus 8 for the entry in the all array. You're allocating 100M of them, which is ~10GB of live data. Go will allocate memory 2x that (with GOGC=100) for its heap, so you will need ~20GB to run. This is the memory usage I see when running in a loop forever. I suspect it fails on your machine on the 3rd iteration just because of how Go grows the heap towards that goal size.

Get a larger machine, or set GOGC to a smaller value. Running runtime.GC explicitly is kinda like setting GOGC to a smaller value, if you can manage to call it at the right time (when the all slice is not live).

I'm going to close, as I don't think there's a bug here. If you can demonstrate otherwise (heap growing to >2x live data), please reopen.

@oghaffarinia
Copy link
Author

Thanks for explaining, calling runtime.GC() solves the problem, and for some reason the automatic garbage collection does not take place for a long running program (even running for several days).
I will update this if I can demonstrate the problem more precisely, thanks anyway

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

5 participants