What processor? What OS?

At least on GNU/Linux the "fatal error: runtime: out of memory" message can only occur when mmap returns ENOMEM.

24-core Linux box. CentOS, kernel 2.6.32-431.29.2.el6.x86_64

From /proc/cpuinfo:

model name : Intel(R) Xeon(R) CPU L5640 @ 2.27GHz

I have the output from free -m and ps from at most one minute before it crashed:

             total       used       free     shared    buffers     cached
Mem:        387592      15402     372190          0        428       6836
-/+ buffers/cache:       8138     379454
Swap:         1023          0       1023

And the output from

    ps fax -ouser,pid,ppid,%cpu,%mem,vsz,rss,tty,stat,start,time,cmd k-vsz | grep -v USER 2>/dev/null;

(trimmed to remove everything other than the process)

dgryski   5685 21823 12.4  1.0 5256760 4130092 pts/1 Sl+    Aug 10 23:27:50      \_ ./cmserver.15rc1 -w=172800 -gp=42013 -p=8080 -e=3600 -prefix=5 -stdout

I suppose I should point out too this ran for months rock solid on 1.4.2.

Are you sure that you're not running out of process/threads (both are accounted the same way on Linux)? In Go 1.5, GOMAXPROCS has changed from 1 to 24 on your Linux box. Could this lead to many additional threads being created when you perform certain kind of operations (especially blocking operations other than sleeping or socket I/O). Do you do any file I/O?

What's the output of ulimit -a? Can you compare the output of ps -eLf with Go 1.4.2 and Go 1.5 to see how many threads there are for your application's PID before/after?

I'm going to see if this still fails with 1.5. I have reason to believe that after ~10 days of traffic, one of the metrics gets enough leaves that the radix tree supporting prefix queries is determined to be "full" and freed, leaving a very large amount of work for the garbage collector to do. If this is related to 3ae1704 , then this patch was not present in rc1 but was present in 1.5 final so the bug should not be occurring. I will build the server with 1.5 final on Monday and report back in a few weeks.

Crashed again with 1.5, after a little more than two weeks. Currently my assumption is that it might be an issue with one of the queries we process, which (for whatever reason), nobody had tried while running 1.4.2.

Closing this for now, as I'm pretty sure this is a dup of #12233 .

For a currently running server:

<dgryski@()memstore[~]> sysctl vm.max_map_count
vm.max_map_count = 65530
<dgryski@()memstore[~]> grep -c ^Size /proc/`pidof cmserver.151`/smaps 
55199