It seems like my change may be genuinely causing us to use significantly more memory on Windows. Particularly, the "fatal error: failed to allocate arena index" buried in there suggests that the machine is actually low on RAM at that point. I'll try running some of the x/benchmarks benchmarks on the gomote since they report their own memory use, and I'll make that panic dump more information about memory consumption and the layout of the heap. @alexbrainman, any suggestions of other good ways to inspect memory usage and memory layout on Windows?

Every failure above and most (though not all) of the failures I've looked at on the dashboard have been mapping the very first heap arena. We haven't even mapped the race shadow for the heap yet. The failing process can't possibly be using much memory itself at this point, so it's almost certainly the other things running concurrently that are using the memory.

It may simply be that TSAN maps 4.5 bytes for every byte we map (MapShadow,
kShadowMultiplier), so each 64MB heap mapping turns into a 352MB mapping. If there are several small processes running that are using significantly less than their initial 64MB mapping, the combined overhead may be too much. Plus, if I'm reading the TSAN code correctly, all 352MB are committed.

I could change Go so that it calls racemapshadow more incrementally as it uses more of each 64MB arena, though this definitely complicates things.

Ping @dvyukov for thoughts. Dmitry, last week I changed the Go memory allocator to map 64MB at a time (always 64MB aligned) and eliminated the assumption that the heap is contiguous in memory. TSAN seems fine with this on Linux, but is running into "out of memory" errors on Windows. I know there are various Go-specific changes to the TSAN runtime. Have I just violated some of its assumptions about the Go heap?

The fact that we reserve and commit 32MB of memory for the arena index may also be contributing to the problem. @alexbrainman, it would be fine to map that memory incrementally, but it would be really nice to have zero-filled memory backing most of it, even if it's read-only. Is there a way to get zeroed memory in Windows that doesn't count against the commit limit?

@alexbrainman, any suggestions of other good ways to inspect memory usage and memory layout on Windows?

Sometimes I use https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap and https://docs.microsoft.com/en-us/sysinternals/downloads/rammap Unfortunately these are GUI apps and they won't work over gomote. But if you need a Windows computer to play with, you can always build one on GCE.

You might be interested to watch

https://www.youtube.com/watch?v=AjTl53I_qzY
and
https://www.youtube.com/watch?v=6KZdNsq1YV4

if you have 3 spare hours in your life.

I also use https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer (not for memory problems), but for debugging in general.

but it would be really nice to have zero-filled memory backing most of it, even if it's read-only. Is there a way to get zeroed memory in Windows that doesn't count against the commit limit?

I do not know. I will try to search old my records.

But you can also search yourself. For example looking at my usual hunting ground (Raymond Chen's blog) - searching for "Virtual­Alloc zero", I get this list

https://social.msdn.microsoft.com/search/en-US?rq=site%3Ablogs.msdn.microsoft.com%2Foldnewthing&rn=oldnewthing&ral=1&query=Virtual%C2%ADAlloc%20zero&ac=5

and https://blogs.msdn.microsoft.com/oldnewthing/20170113-00/?p=95185 on that list looks interesting. But ultimately you should try your ideas and see what happens with VMMap.

I hope it helps.

Alex

I will try to search old my records.

I did. There is nothing much there:

https://blogs.msdn.microsoft.com/oldnewthing/20170317-00/?p=95755
http://aeops.blogspot.com.au/2009/10/art-of-memreset-flag.html

Maybe @dvyukov can also help with your question. He does know a bit about Windows, but he is shy.

Alex

MapShadow assumes that Go heap is continuous and grows up. See the mapped_meta_end variable in MapShadow. Tsan also has pretty tight assumptions about layout of virtual address space:
https://github.com/llvm-mirror/compiler-rt/blob/409d44eac7a558f5de389599503db213044819b7/lib/tsan/rtl/tsan_platform.h#L386

That MapShadow complexity was actually imposed by Windows. On linux/bsd we simply map 128TB of memory at start and we are done. But Windows eagerly allocates page tables (potentially even for reserved memory) and charges the process for it. So if we map 128TB and page tables take, say, 1/1000-th, Windows will try to eagerly allocate 128GB which always fails. So we had to do this lazy shadow allocation. There is additional problem with the "meta" shadow which is 1/2 of application memory, so even if an app allocates in dwAllocationGranularity chunks, the shadow is smaller than dwAllocationGranularity.

I've looked at a dozen of failures on the build dashboard and it seems that the broken assumption about continuous heap is not the problem (at least not yet) and they simply fail due to OOM errors:

==2424==ERROR: ThreadSanitizer failed to allocate 0x000010000000 (268435456) bytes at 0x040000000000 (error code: 1455)
==2684==ERROR: ThreadSanitizer failed to allocate 0x000002000000 (33554432) bytes at 0x07c000000000 (error code: 1455)

These are allocations of memory shadow and meta shadow of expected sizes at expected addresses.

Is it possible to reduce 64MB const under race detector on Windows?
Note that this is also penalizes programs without race to some degree. Overallocation of unused memory is not completely free on Windows.

@alexbrainman, thanks for the pointers. I'd looked over the MSDN docs for virtual memory functions and didn't get much out of it, but those gave me some ideas I can at least try, though it may just not be possible to do what I want.

Tsan also has pretty tight assumptions about layout of virtual address space

Thanks for pointing me to the memory map. Go will still grow the heap contiguously from 0xc000000000 unless it runs into some other mapping. If it does, it will jump to 0xc100000000 and so on. It looks like TSAN is okay with that from the memory layout standpoint, but that MapShadow will try to map a huge amount of meta shadow to cover the 4GB jump. It looks like the mapping of the regular shadow will be fine; does it have to map all of that in-between space for the meta shadow?

If Go falls all the way back to 0xe000000000, it will ask the OS for whatever it can give, but something would have to go pretty wrong to get to that point.

Is it possible to reduce 64MB const under race detector on Windows?

Unfortunately, that would increase the size of the arena index proportionally, and it's already 32MB. That said,

1. Since that 32MB of arena index is committed on Windows, it may already be a problem and maybe I need to add some indirection there. If I add the indirection, then maybe it's okay to reduce the arena size and make the arena index bigger.

2. Since we consume memory within each 64MB arena sequentially, I could commit the memory backing each arena more incrementally (both for the OS and for TSAN).

It looks like the mapping of the regular shadow will be fine; does it have to map all of that in-between space for the meta shadow?

No, it's just am implementation detail to deal with dwAllocationGranularity for the meta shadow.
The only thing that matters in the end is that shadow for regions that Go program will actually touch is mapped.

Since we consume memory within each 64MB arena sequentially, I could commit the memory backing each arena more incrementally (both for the OS and for TSAN).

Sounds like this can work.
If Go runtime promises to map heap by at least aligned 128KB blocks, then MapShadow can be significantly simplified. I.e. we could them map both shadows with a single VirtualAlloc without accounting for alignment, previous allocations, etc.

Change https://golang.org/cl/96780 mentions this issue: runtime: reduce arena size to 4MB on 64-bit Windows

Change https://golang.org/cl/96779 mentions this issue: runtime: support a two-level arena map

If Go runtime promises to map heap by at least aligned 128KB blocks, then MapShadow can be significantly simplified. I.e. we could them map both shadows with a single VirtualAlloc without accounting for alignment, previous allocations, etc.

That would be great. We can definitely guarantee 128KB alignment. In fact, it's 64MB-aligned right now, and will still be 4MB-aligned after CL 96780. Should I file a bug to track that, and should I do it here or is there a better place to file it?

This tracker is good (add label:RaceDetector).

Done: #24133.