This is intended to be explained in https://tip.golang.org/doc/gc-guide#GOGC, though that section has a lot more words and could perhaps have a more direct, succinct mention of the total heap size.

cc @mknyszek

We thought that the garbage collector returns unused heap memory back to the operating system, which was wrong.

To be clear, it does return unused heap memory to the OS, but it's not going to return memory to the OS it thinks it's going to use in the near future. It generally retains enough for the application to allocate up to the next heap goal (runtime.MemStats.NextGC, or the runtime/metrics metric /gc/heap/goal:bytes) without the OS having to page in new physical memory, which is expensive.

I think (but am not certain) that this is what you're seeing.

"Since the Go garbage collector is designed to retain about twice as much memory as your live heap size, debug.FreeOSMemory is expected to free about half of your memory"

This quote is true, but there's a lot of nuance to this. If your application is allocating memory regularly, debug.FreeOSMemory may appear to have little to no effect because the result will be so short-lived. As Austin notes in the comment you linked, an active application can quickly end up using that memory again. It makes a lot more sense when you expect your application to be idle for a long time. Simultaneously, calling debug.FreeOSMemory can incur a substantial CPU cost. If your application is active, then memory that was just released to the OS is likely to get paged in immediately, which as I mentioned above is expensive. It also forces a full GC cycle, increasing GC frequency.

debug.FreeOSMemory should probably be in the GC guide, but I want to avoid recommending debug.FreeOSMemory as much as possible because it's really a last resort.

This is intended to be explained in https://tip.golang.org/doc/gc-guide#GOGC, though that section has a lot more words and could perhaps have a more direct, succinct mention of the total heap size.

FWIW, the total heap size is defined in a single sentence toward the top of that section. Re-reading that section, most of the text appears to be footnotes and examples (that is, "it can be used to turn off the GC," "Go 1.18 did..., "here's an example," "here's how to use the interactive example"). Grouping all that together does tend to make one's eyes glaze over, I admit. We could probably make this a lot better by moving the footnotes into actual footnotes and putting the examples under an "examples" subheading. IMO the definition itself is OK if you stop reading after the first few sentences.

This doesn't really help with the "eyes glazing over" problem, but I think maybe what's missing here is a short sentence like "the GC always retains enough memory for the application to allocate up to the target heap size." Perhaps if we restructure the section the extra sentence won't be so bad.

My application allocates memory on demand, e.g.when it receives a web request. The annoying part was, that it does not return all used memory back to the OS regardless how long we wait. So the memory usage (from OS point of view) did not go back to the level that we had before the web request.

The exact amount which gets returned is not so important to me, but the facts that it does not return all free memory. To be honest, I understood the gc-guide only partially. Maybe that's the reason why the final result from OS point of view was not clear to me.

Anyway, I ensured already that the memory usage does not constantly increase over time. This works fine.

The annoying part was, that it does not return all used memory back to the OS regardless how long we wait.

Out of curiosity, how long did you wait? There's a forced GC every 2 minutes if it hasn't run sooner. Once that happens, the background scavenger should kick in and return memory back to the OS gradually.

I waited 2 days.

Maybe the attached diagram is helpful.

I think the graph shows the same as the "RES" column of the top command.

Two instances of the module had been deployed yesterday. They consumed the same amount of memory initially. After some load, the green pod was constantly using more memory than the yellow one, even after a whole night of very few activity.

Today morning I produced some load at 07:35 - 07:38. You can see that the memory usage of both pods did not go back down to the original level. That is what we first interpreted as a memory leak.

I am already 80% sure that we have no real memory leak, because a few days ago I created a lot load for two hours where the memory usage was high but did not constantly increase. My boss is 95% sure 😃

Metrics of the green POD collected at 08:45 (not sure if they are helpful):

go_gc_duration_seconds{quantile="0"} 7.4805e-05
go_gc_duration_seconds{quantile="0.25"} 0.000113463
go_gc_duration_seconds{quantile="0.5"} 0.000141882
go_gc_duration_seconds{quantile="0.75"} 0.000211009
go_gc_duration_seconds{quantile="1"} 0.086942859
go_gc_duration_seconds_sum 0.379772665
go_gc_duration_seconds_count 615
go_goroutines 562
go_info{version="go1.21.5"} 1
go_memstats_alloc_bytes 5.6752192e+07
go_memstats_alloc_bytes_total 1.3669962256e+10
go_memstats_buck_hash_sys_bytes 1.633549e+06
go_memstats_frees_total 1.76503274e+08
go_memstats_gc_sys_bytes 6.688632e+06
go_memstats_heap_alloc_bytes 5.6752192e+07
go_memstats_heap_idle_bytes 3.3284096e+07
go_memstats_heap_inuse_bytes 6.6985984e+07
go_memstats_heap_objects 408825
go_memstats_heap_released_bytes 2.1897216e+07
go_memstats_heap_sys_bytes 1.0027008e+08
go_memstats_last_gc_time_seconds 1.7048726598582582e+09
go_memstats_lookups_total 0
go_memstats_mallocs_total 1.76912099e+08
go_memstats_mcache_inuse_bytes 31200
go_memstats_mcache_sys_bytes 31200
go_memstats_mspan_inuse_bytes 1.108128e+06
go_memstats_mspan_sys_bytes 1.450344e+06
go_memstats_next_gc_bytes 8.9540592e+07
go_memstats_other_sys_bytes 4.704587e+06
go_memstats_stack_inuse_bytes 4.58752e+06
go_memstats_stack_sys_bytes 4.58752e+06
go_memstats_sys_bytes 1.19365912e+08
go_threads 30
process_cpu_seconds_total 3805.21
process_max_fds 1.048576e+06
process_open_fds 145
process_resident_memory_bytes 1.21344e+08
process_start_time_seconds 1.70480085494e+09
process_virtual_memory_bytes 1.365942272e+09
process_virtual_memory_max_bytes 1.8446744073709552e+19
promhttp_metric_handler_requests_in_flight 1
promhttp_metric_handler_requests_total{code="200"} 2396
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0