#14521 Maybe this is relevant. But my linux kernel is 2.6. I am not sure if the madvise system call will use madvdontneed, I seted the evn 'madvdontneed=1'

Another thing:

proc.go sysmon() 
 // If a heap span goes unused for 5 minutes after a garbage collection,
// we hand it back to the operating system.

MemStats reported the heapIdle is 52234674176, And it will not decrease when I periodic call debug.FreeOSMemory(). But how to know if these spans are idle for no more than 5 minutes ?
Any help, thx

if errno := madvise(v, n, int32(advise)); advise == _MADV_FREE && errno != 0 {
		// MADV_FREE was added in Linux 4.5. Fall back to MADV_DONTNEED if it is
		// not supported.
		atomic.Store(&adviseUnused, _MADV_DONTNEED)
		madvise(v, n, _MADV_DONTNEED)
	}

This code shows that my linux will only use MADV_DONTNEED.

From #14521 (comment), it seems like you should be looking at HeapReleased, rather than HeapIdle. HeapReleased is increasing in your example, so it appears to be released to linux periodically as you would expect.

/cc @aclements @randall77 since they have more context here, and may better understand the issue.

I believe we are seeing something similar to this in kubernetes after upgrading to Go 1.12.

Our memory usage never decreases, causing our pods to report 80-90% memory utilization after handling a large request. Even though internally, the go-runtime thinks it is only holding onto 10-20%.

Setting the GODEBUG=madvdontneed=1 fixes this and memory is returned to the pod.

Go version: 1.12.9
GOOS: linux
GOARCH: amd64

Setting the GODEBUG=madvdontneed=1 fixes this and memory is returned to the pod.

That's more of a workaround than a fix.

The actual fix (if the goal is to use memory-based autoscaling, or memory-based alerting) is to not consider the memory that the runtime has marked as free. Keep in mind that the OS can reuse the memory that the runtime has marked as free - but RSS does not capture this information.

The fact that the RSS is not going down means that the OS actually does not need that memory right now, so it leaves it in the RS of the go process. If the OS needed that memory, the RSS would go down (that's why I said this only matters for autoscaling or alerting, because for other purposes that memory has effectively been returned to the OS).

If you want to do this in a language-agnostic way (keep in mind that go is not the only one that uses MADV_FREE: node and java[1] do as well) is to subtract LazyFree from the RSS.

Agree though that It may still make sense to eventually completely deallocate if memory has been completely idle for an extended period of time, as this may be useful e.g. if you run multiple processes in a mem cgroup.

[1]: java uses MADV_FREE in some (but not all) configurations

__

Setting the GODEBUG=madvdontneed=1 fixes this and memory is returned to the pod.

That's more of a workaround than a fix.

The actual fix (if the goal is to use memory-based autoscaling, or memory-based alerting) is to not consider the memory that the runtime has marked as free. Keep in mind that the OS can reuse the memory that the runtime has marked as free - but RSS does not capture this information.

The fact that the RSS is not going down means that the OS actually does not need that memory right now, so it leaves it in the RS of the go process. If the OS needed that memory, the RSS would go down (that's why I said this only matters for autoscaling or alerting, because for other purposes that memory has effectively been returned to the OS).

If you want to do this in a language-agnostic way (keep in mind that go is not the only one that uses MADV_FREE: node and java[1] do as well) is to subtract LazyFree from the RSS.

Agree though that It may still make sense to eventually completely deallocate if memory has been completely idle for an extended period of time, as this may be useful e.g. if you run multiple processes in a mem cgroup.

[1]: java uses MADV_FREE in some (but not all) configurations

RSS is not going down, but my contianer is oom in kubernetes. Mean OS memory is the host machine?

RSS is not going down, but my contianer is oom in kubernetes. Mean OS memory is the host machine?

If OOM happens it means one of two things:

1. the go process is trying to allocate more memory than the one allowed by the cgroup (a.k.a. limits in your k8s manifest) - either you have set a limit that is too low, or you have a leak somewhere
2. the host memory is overcommitted

Both cases have nothing to do with this issue though, or with go in general.

Same here, the application memory usage never decreases.After test in three different machines for 20 minutes.The app which running on macOS gc works properly .

I think there are too many things going on in this issue.

As to the original issue, @Anteoy , HeapIdle is expected to increase monotonically. It's not measuring OS memory held by Go. The documentation says:

	// Idle spans have no objects in them. These spans could be
	// (and may already have been) returned to the OS, or they can
	// be reused for heap allocations, or they can be reused as
	// stack memory.

Note the "could be returned to the OS". As @katiehockman pointed out, HeapReleased is the subset of HeapIdle that has been released to the OS. (I realize having all of these MemStats that are subsets of other MemStats is pretty annoying. We're working on a new API that will hopefully be much better structured.)

Everyone else is seeing a completely different issue. @CAFxX summarized it very well. Go uses MADV_FREE if the kernel supports it to release memory because it's dramatically more efficient than MADV_DONTNEED. A lot of modern memory managers do this. This has the rather annoying consequence that RSS doesn't go down unless the OS is actually under memory pressure (I'm not sure why Linux chose to do it this way). As @CAFxX mentioned, you can subtract LazyFree from the RSS to account for this, though top/htop/etc don't do this. Container memory limits do account for this, so if your process is OOMing, it's because it's just using too much memory, and not related to MADV_FREE.

I think the explanation about HeapIdle resolves the original issue here. See #42330 (a meta-issue I just posted) for reverting to MADV_DONTNEED by default on Linux because of the user experience issues discussed on this issue. I think that takes care of everything, so I'm going to close this, but please reply if that's not the case.

Change https://golang.org/cl/267100 mentions this issue: runtime: default to MADV_DONTNEED on Linux