Change https://golang.org/cl/269998 mentions this issue: runtime: add force_small_arena build tag

CC @aclements @mknyszek

I don't think a build tag (or any optional behavior) is the right way to resolve running up against strict overcommit. It's another configuration to maintain.

I think we should consider doing a 64 MiB reservation (PROT_NONE, which doesn't count toward overcommit if it hasn't been touched before) and mapping pages read-write as needed. We already have a mechanism in the runtime to "grow" into an existing arena (see (*mheap).grow and mheap.curArena) we would just need to move the sysMap call in (*mheap).sysAlloc there instead. Then we get this behavior on every platform which could have similar restrictions for 64 MiB arenas.

Such a change does mean we can retain the performance benefits of having a shallower arena index on 64-bit platforms (1-2%), but where we lose is that we potentially have to do more syscalls. I personally don't think that's a big deal because heap growths are relatively rare whereas we access the arena index extremely often. Notably, we won't have to do more syscalls than on 4 MiB arena platforms since we also already round up ask in (*mheap).grow to pallocChunkPages (which happens to be 4 MiB).

It's a fairly small and easy change to make, though unfortunately I think at this point it will have to wait for Go 1.17, unless I'm wrong.

@aclements WDYT?

I agree that I don't want to ship a configuration flag for this. But I like @mknyszek's idea of using mheap.curArena to map the reservation more incrementally. This would also solve #28081 (see in particular #28081 (comment)). I think we have to be careful not to map it in too small of chunks just to reduce mmap overhead, but a few megs at a time ought to be plenty.

I think this is too late for 1.16, unfortunately, but since it should be a pretty simple change we could make it now and queue it up for 1.17. @mknyszek , care to do the honors?

(Actually, we should definitely do it at least 2 MiB at a time so we don't break up huge pages on amd64.)

Change https://golang.org/cl/270537 mentions this issue: runtime: prepare arenas for use incrementally

@calder if it's not too much trouble, do you mind trying out https://golang.org/cl/270537 and seeing if it resolves issues with strict overcommit for you?