Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak, the memory occupied by waiting several hours will not be returned to the operating system #42455

Closed
franklwel1990 opened this issue Nov 9, 2020 · 3 comments

Comments

@franklwel1990
Copy link

franklwel1990 commented Nov 9, 2020

What version of Go are you using (go version)?

go version go1.15.3 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/jhw/www/html/gopath/pkg/mod"
GONOPROXY="gitlab.com/idmabar,bitbucket.org/idmabar,github.com/idmabar"
GONOSUMDB="gitlab.com/idmabar,bitbucket.org/idmabar,github.com/idmabar"
GOOS="linux"
GOPATH="/jhw/www/html/gopath"
GOPRIVATE="gitlab.com/idmabar,bitbucket.org/idmabar,github.com/idmabar"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/jhw/www/html/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/jhw/www/html/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build434819960=/tmp/go-build -gno-record-gcc-switches"

What did you do?

通过go-xorm/xorm读取500万行数据,占用linux的内存used一直不会被释放

code:
d := make([]TpccTestingDetail, 0, 5000000)
err := app.DB.Table(req.Table).Select("s_i_id,s_w_id,s_quantity,s_dist_01,s_dist_02,s_dist_03,s_dist_04,s_dist_05,s_dist_06,s_dist_07,s_dist_08,s_dist_09,s_dist_10,s_ytd,s_order_cnt,s_remote_cnt,s_data").Limit(5000000, 0).Find(&d)

sql查询后(大小是kb):
Alloc = 5533008 TotalAlloc = 3285136 Sys = 8489336 HeapReleased = 52560 HeapAlloc = 5533008 HeapSys = 7994656 HeapIdle = 1383728 HeapInuse = 6610928 NumGC = 15

手动runtime.GC()后(大小是kb):
Alloc = 983 TotalAlloc = 3285136 Sys = 8489336 HeapReleased = 52528 HeapAlloc = 983 HeapSys = 7994656 HeapIdle = 7992224 HeapInuse = 2432 NumGC = 16

手动debug.FreeOSMemory()后(大小是kb):
Alloc = 984 TotalAlloc = 32851366 Sys = 8489336 HeapReleased = 7992088 HeapAlloc = 984 HeapSys = 7994656 HeapIdle = 7992240 HeapInuse = 2416 NumGC = 17

go进程(ps -p 2446):
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
2446 pts/3 Sl+ 0:01 4 5397 2044214 32752 0.1 go-test

系统free -m 后:
total used free shared buff/cache available
Mem: 16013 13501 309 35 2202 9903
Swap: 0 0 0

What did you expect to see?

能正常释放应用程序占用的内存归还操作系统。(函数返回,操作完空闲一段时间后,手动释放)

What did you see instead?

持续观察linux的used一直没有归还操作系统,直到进程被kill掉
linux使用的内存 used 观察几个小时后都没有变化。

@fanlv
Copy link

fanlv commented Nov 9, 2020

go 1.12 版本以后申请内存是用的MADV_FREE,只有操作系统内存不够的时候(使用docker的话可能会导致这个判断不准确),才会回收。具体可以自己google下。 你可以设置 GODEBUG=madvdontneed=1 取消这个优化,在测试下。

@martisch
Copy link
Contributor

martisch commented Nov 9, 2020

See #42330. Linux also needs to request the memory back e.g. under memory pressure before stats change.

@mvdan
Copy link
Member

mvdan commented Nov 9, 2020

What @martisch said. Also, see https://golang.org/wiki/Questions.

@mvdan mvdan closed this as completed Nov 9, 2020
@golang golang locked and limited conversation to collaborators Nov 9, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants