runtime: Memory not returned to system on linux #37585

TimmyOVO · 2020-02-29T17:17:38Z

What version of Go are you using (`go version`)?

$ go version
go version go1.13.3 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/lib/golang"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build842407928=/tmp/go-build -gno-record-gcc-switches"

What did you do?


import (
	"runtime/debug"
	"time"
)

func main() {
	bytes := make([]byte, 326766666)
	for k, _ := range bytes {
		bytes[k] = 0
	}
	bytes = nil
	go func() {
		time.Sleep(time.Minute * 5)
		println("after 5 minutes")
	}()
	time.Sleep(time.Minute * 20)
}

What did you expect to see?

memory should be returned to system

What did you see instead?

On macos the memory be returned to system after about 3 minutes, but after waiting for 20 minutes, the memory still being use by the application on linux

The text was updated successfully, but these errors were encountered:

randall77 · 2020-02-29T17:36:50Z

Does using runtime/debug.FreeOSMemory help?
Does Go 1.14 help?

What method are you using to measure RSS? A common confusion is that while Go releases memory back to the OS, if the OS has no other demand for that memory it won't actually take the pages back. Only if there are other demands for that memory will the Go process's RSS go down.

Please don't post screenshots of code or logs. It makes it hard for us to reproduce your results. Just copy-paste the text directly.

TimmyOVO · 2020-02-29T18:15:55Z

update Golang to 1.14 doesn't help
using debug.FreeOSMemory either
but after downgrade Golang to 1.11 works

the code I'm using for test

package main

import (
	"runtime/debug"
	"time"
)

func main() {
	bytes := make([]byte, 326766666)
	for k, _ := range bytes {
		bytes[k] = 0
	}
	bytes = nil
	go func() {
		time.Sleep(time.Minute * 5)
		println("after 5 minutes")
		debug.FreeOSMemory()
	}()
	time.Sleep(time.Minute * 20)
}

Compiled by Golang 1.11.13

go version go1.11.13 darwin/amd64

thepudds · 2020-02-29T18:56:56Z

@TimmyOVO Are you on a relatively recent Linux?

If so, what does this show for the PID of your process:

grep LazyFree /proc/<PID>/smaps | grep -v " 0 kB"

From /proc man page:

LazyFree %lu (since Linux 4.12)
          Shows the amount of memory marked by madvise(2) MADV_FREE.

From Go 1.12 release notes:

On Linux, the runtime now uses MADV_FREE to release unused memory. This is more efficient but may result in higher reported RSS. The kernel will reclaim the unused data when it is needed.

(On mobile, so brief).

thepudds · 2020-02-29T23:11:12Z

I updated @TimmyOVO's code from above (#37585 (comment)) to periodically output RSS memory values as reported by ps and top, as well as to output the sum of LazyFree values from /proc/<pid>/smaps.

Full update is on playground, but the primary addition is this quick & dirty function:

reportMem := func(header string) {
	log.Println("---", header, "---")
	pid := os.Getpid()
	shell("ps rss:  ", `ps -o rss= -p %d`, pid)
	shell("top res: ", `top -b -n 1 | awk '/^[ ]*%d[ ]/ {print $6}'`, pid)
	shell("LazyFree:", `cat /proc/%d/smaps | awk '/LazyFree/ {sum+= $2} END {print sum}'`, pid)
}

I think the results are as expected for Go 1.11, 1.13, and 1.14, at least as far as I understand things in this general area:

In 1.11, RSS declines significantly after FreeOSMemory.
In 1.13 and 1.14, after 5 minutes RSS is still high while LazyFree has climbed to almost the same value as RSS (>300MB), and the subsequent FreeOSMemory does not seem to have material impact.

Go 1.11

$ go1.11.10 run main.go 
2020/02/29 22:33:01 --- before alloc ---
2020/02/29 22:33:01 ps rss:   1544
2020/02/29 22:33:02 top res:  1544
2020/02/29 22:33:02 LazyFree: 0
2020/02/29 22:33:02 --- after alloc ---
2020/02/29 22:33:02 ps rss:   331076
2020/02/29 22:33:02 top res:  331076
2020/02/29 22:33:02 LazyFree: 0
2020/02/29 22:38:02 --- after 5 minutes --- 
2020/02/29 22:38:02 ps rss:   331076
2020/02/29 22:38:02 top res:  331076
2020/02/29 22:38:02 LazyFree: 0
2020/02/29 22:38:02 --- after FreeOSMemory ---
2020/02/29 22:38:02 ps rss:   11908
2020/02/29 22:38:03 top res:  11908
2020/02/29 22:38:03 LazyFree: 0

Go 1.13

go1.13.3 run main.go
2020/02/29 22:40:05 --- before alloc ---
2020/02/29 22:40:05 ps rss:   1572
2020/02/29 22:40:06 top res:  1572
2020/02/29 22:40:06 LazyFree: 0
2020/02/29 22:40:06 --- after alloc ---
2020/02/29 22:40:06 ps rss:   331320
2020/02/29 22:40:06 top res:  331320
2020/02/29 22:40:06 LazyFree: 2384
2020/02/29 22:45:06 --- after 5 minutes ---
2020/02/29 22:45:06 ps rss:   331320
2020/02/29 22:45:06 top res:  331320
2020/02/29 22:45:06 LazyFree: 319104
2020/02/29 22:45:06 --- after FreeOSMemory ---
2020/02/29 22:45:06 ps rss:   331320
2020/02/29 22:45:06 top res:  331320
2020/02/29 22:45:06 LazyFree: 319032

Go 1.14

$ go1.14 run main.go 
2020/02/29 22:47:13 --- before alloc ---
2020/02/29 22:47:13 ps rss:   1852
2020/02/29 22:47:14 top res:  1852
2020/02/29 22:47:14 LazyFree: 0
2020/02/29 22:47:14 --- after alloc ---
2020/02/29 22:47:14 ps rss:   331312
2020/02/29 22:47:14 top res:  331312
2020/02/29 22:47:14 LazyFree: 0
2020/02/29 22:52:14 --- after 5 minutes ---
2020/02/29 22:52:14 ps rss:   331312
2020/02/29 22:52:14 top res:  331312
2020/02/29 22:52:14 LazyFree: 319352
2020/02/29 22:52:14 --- after FreeOSMemory ---
2020/02/29 22:52:14 ps rss:   331576
2020/02/29 22:52:14 top res:  331576
2020/02/29 22:52:14 LazyFree: 319372

This is on Ubuntu:

$ grep DESC /etc/lsb-release
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
$ uname -a
Linux instance-fzgo-test-7 4.15.0-1037-gcp #39-Ubuntu SMP Wed Jul 3 06:28:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

bonedaddy · 2020-05-10T03:11:14Z

If you run your binary with GODEBUG=madvdontneed=1 memory will be returned to the OS

gopherbot · 2020-11-02T00:40:39Z

Change https://golang.org/cl/267100 mentions this issue: runtime: default to MADV_DONTNEED on Linux

In Go 1.12, we changed the runtime to use MADV_FREE when available on Linux (falling back to MADV_DONTNEED) in CL 135395 to address issue #23687. While MADV_FREE is somewhat faster than MADV_DONTNEED, it doesn't affect many of the statistics that MADV_DONTNEED does until the memory is actually reclaimed under OS memory pressure. This generally leads to poor user experience, like confusing stats in top and other monitoring tools; and bad integration with management systems that respond to memory usage. We've seen numerous issues about this user experience, including #41818, #39295, #37585, #33376, and #30904, many questions on Go mailing lists, and requests for mechanisms to change this behavior at run-time, such as #40870. There are also issues that may be a result of this, but root-causing it can be difficult, such as #41444 and #39174. And there's some evidence it may even be incompatible with Android's process management in #37569. This CL changes the default to prefer MADV_DONTNEED over MADV_FREE, to favor user-friendliness and minimal surprise over performance. I think it's become clear that Linux's implementation of MADV_FREE ultimately doesn't meet our needs. We've also made many improvements to the scavenger since Go 1.12. In particular, it is now far more prompt and it is self-paced, so it will simply trickle memory back to the system a little more slowly with this change. This can still be overridden by setting GODEBUG=madvdontneed=0. Fixes #42330 (meta-issue). Fixes #41818, #39295, #37585, #33376, #30904 (many of which were already closed as "working as intended"). Change-Id: Ib6aa7f2dc8419b32516cc5a5fc402faf576c92e4 Reviewed-on: https://go-review.googlesource.com/c/go/+/267100 Trust: Austin Clements <austin@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com>

changkun · 2021-01-05T09:28:39Z

Because 05e6d28 is landed on the master, this issue can be closed.

odeke-em · 2021-01-05T09:35:43Z

Thank you @changkun for the notice, and @aclements for the fix, also happy New Year everyone!

dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 2, 2020

dmitshur added this to the Backlog milestone Mar 2, 2020

dmitshur added the OS-Linux label Mar 2, 2020

JensErat mentioned this issue Jul 29, 2020

Account and export LazyFree memory by container google/cadvisor#2634

Open

aclements mentioned this issue Nov 2, 2020

runtime: default to MADV_DONTNEED on Linux #42330

Closed

odeke-em closed this as completed Jan 5, 2021

AlexiaChen mentioned this issue Apr 30, 2021

记解决一件奇怪的线上golang的"OOM"问题 AlexiaChen/AlexiaChen.github.io#138

Open

geekhckr mentioned this issue Jun 22, 2021

Possible memory leak involving topdown/http? open-policy-agent/opa#2978

Closed

golang locked and limited conversation to collaborators Jan 5, 2022

gopherbot added the FrozenDueToAge label Jan 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: Memory not returned to system on linux #37585

runtime: Memory not returned to system on linux #37585

TimmyOVO commented Feb 29, 2020 •

edited

randall77 commented Feb 29, 2020 •

edited

TimmyOVO commented Feb 29, 2020 •

edited by randall77

thepudds commented Feb 29, 2020 •

edited

thepudds commented Feb 29, 2020

bonedaddy commented May 10, 2020

gopherbot commented Nov 2, 2020

changkun commented Jan 5, 2021

odeke-em commented Jan 5, 2021

runtime: Memory not returned to system on linux #37585

runtime: Memory not returned to system on linux #37585

Comments

TimmyOVO commented Feb 29, 2020 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

randall77 commented Feb 29, 2020 • edited

TimmyOVO commented Feb 29, 2020 • edited by randall77

thepudds commented Feb 29, 2020 • edited

thepudds commented Feb 29, 2020

Go 1.11

Go 1.13

Go 1.14

bonedaddy commented May 10, 2020

gopherbot commented Nov 2, 2020

changkun commented Jan 5, 2021

odeke-em commented Jan 5, 2021

TimmyOVO commented Feb 29, 2020 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

randall77 commented Feb 29, 2020 •

edited

TimmyOVO commented Feb 29, 2020 •

edited by randall77

thepudds commented Feb 29, 2020 •

edited