What did you do?
I changed runtime/malloc.go to set _MHeapMap_TotalBits=42 to support 4TB of ram.

You made a modification in the runtime and it's causing GC crashes. I'm, hum... not sure that this is a valid bug report.

But maybe your patch is correct and it exposed a real issue, so cc @aclements in case he wants to investigate.

Well, this clearly isn't a supported configuration, but I'm happy to help debug. :) In principle it could work, but I'm not particularly surprised that it doesn't.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf1 pc=0x454fcb]

runtime stack:
runtime.throw(0x51855c, 0x2a)
        /users/jjay/src/github.com/golang/go/src/runtime/panic.go:605 +0x95
runtime.sigpanic()
        /users/jjay/src/github.com/golang/go/src/runtime/signal_unix.go:351 +0x2b8
runtime.sysMunmap(0x2b494827f000, 0x5c5e438, 0x2b494827f000, 0x2b497b8be000, 0x8a8d650, 0xe1617ddd40, 0x41407e, 0x2b494827f000, 0x5c5e438, 0xe1
617ddde0, ...)
        /users/jjay/src/github.com/golang/go/src/runtime/sys_linux_amd64.s:403 +0x1b

This isn't a double-free (munmap has no such concept), but the munmap syscall is failing. The runtime doesn't expect it to fail, so it intentionally crashes the program at a low level. Probably we should propagate that up and crash with a nicer message, but the real question is why would munmap fail? The address is aligned and the length is sane. The only other failure mode I know off the top of my head is running out of VMAs.

If you could get the error code from munmap, it might shed some light on the problem. Probably just tweak sysMunmap to return the result of the syscall unconditionally and then do the equivalent error check where it's called from Go code and print out the error code before throwing.

If it is the number of VMAs, you could just keep an eye the number of lines in /proc/PID/maps. If that exceeds the value of sysctl vm.max_map_count, then that's what's causing the failure. In this case it's still an interesting question as to why we're winding up with so many mappings, but you can just set vm.max_map_count higher to work around it.

Thanks for the pointer @aclements - that definitely seems to be the issue. vm.max_map_count = 65530 , and I looped printing the line count of /proc/PID/maps and it jumps from 29 to 56119 in under 5s, approximately just before the crash normally occurs (of course it doesn't crash when doing this /s).

Seems to be a one-time spike as the growth is fairly slow before and after this point. The map file basically contains 50k x 2MB regions, so 100GB worth of allocations in 5s. Probably would have crashed most systems... :) I'm guessing the GC takes a bit longer to recover all that, since adding any sort of GODEBUG or tweaking GOGC seems so mitigate the issue (GOGC=30 has 2900 maps at 70% complete, default crashes before 1%).

I'll continue debugging to see if I can find a way to reproduce consistently.

The map file basically contains 50k x 2MB regions, so 100GB worth of allocations in 5s.

Interesting. This happens to be the worst-case behavior of the scavenger on Linux. Take a look at the big comment in runtime/mem_linux.go:sysUnused for an explanation. It's not exactly that there were 100GB of allocations in 5s. Those allocations already happened. Then when the scavenger ran and returned unused memory to the OS, the runtime wound up fragmenting the flags on that range of the address space, which split it into the separate 2MB VMAs you're seeing.

If you don't care about returning memory to the OS (it'll still be reused internally by Go), you could disable the scavenger by just returning from mheap.scavenge in mheap.go.

Such an interesting rabbit hole, I haven't been this deep on system stuff in years... So malloc fails because it can't resize it's span list, which fails because there are too many VMAs, which are too many because the kernel hasn't cleaned up the VMAs, which were released by the scavenger on a recent GC. (and my code is terrible because it made 100GB of temporary allocations!)

For a cluster environment like ours, my code generally the only thing running on a large-memory node so there's no real reason to keep the scavenger enabled. It'd be cool to do this at runtime (I see that GODEBUG=scavenge=2 would be trivial to add), but I'm sure this is not a common use case.

Thanks so much for the help debugging and for the workaround!

which are too many because the kernel hasn't cleaned up the VMAs, which were released by the scavenger on a recent GC.

It's actually a little subtler than this. The VMAs weren't released, and there's nothing for the kernel to clean up. The way the Go runtime releases memory releases the physical memory without releasing the virtual address space, so it actually doesn't affect the VMAs (which map virtual address space). But to avoid a quirk of Linux transparent huge page support, we change the flags on the virtual address space at a 2 MB granularity. Prior to this, the entire Go heap is represented by a few huge VMAs. But the flag changes force the kernel to break up these large VMAs into smaller VMAs since a VMA only has a single set of flags associated with it. In the worst case, you get a checkerboard of flags with distinct VMA covering every 2 MB region.

Thanks so much for the help debugging and for the workaround!

Thanks for the interesting problem. :)

I'm guessing we may have to solve this for real one day, but for now, since it's a niche problem and we've identified a few workarounds, I'm going to go ahead and close this issue.

Just following up, Go tip (to be released as Go 1.11) no longer has any direct limit on the heap size.

The VMA count is likely to still be a problem for huge heaps.