Heavily-allocating and highly-parallel applications' memory allocation latencies regressed significantly in the Go 1.12 release (up to 3-4x in k8s' 99th percentile latency).

Much of the background and work done for this issue has been summarized in kubernetes/kubernetes#75833.

To re-summarize:

The offending commit with bisection was found to be 8e093e7, which introduced the notion of scavenging to make up for the RSS increase for scavenging from scavenged memory (a mouthful). This change landed to fix an issue for some users where, as a result of previous changes in Go 1.12, the allocator could outrun things like debug.FreeOSMemory.

However, these extra syscalls caused an unacceptable increase in the latency of memory allocations which took this slow path which had the heap lock acquired. In the case of an already heavily-allocating and highly-parallel application (like the k8s situation: hundreds of thousands of goroutines and 64 Ps), this can have serious consequences in terms latency because subsequent span allocations' would end up spending a lot of time trying to acquire the heap lock.

While the fundamental issue is more the fact that the span allocator is not very scalable (all span allocations are serialized via a single lock), this is much more difficult to fix. At the very least we should bring back the latency to Go 1.11 or better in these situations.

In investigating this further, we discovered that the fact that the allocator could outrun things like debug.FreeOSMemory was partially because of the VSS growth introduced by the regression in #31616 in Go 1.12, so fixing this bug could be as simple as reverting 8e093e7 and fixing #31616.