Comment 1:

It looks like virtual memory. Try running 50 copies of the program and see if it uses
25GB of RAM or <1GB.
See also issue #5236.

Comment 2:

does this happen on other windows version?
I tested on windows server 2008 R2, but failed to reproduce.

Comment 3 by sjbogdan:

8 running processes ate 5 GB of swap / commit ( see attachment )
Can't test on other systems, attaching executable ( note : 64-bit executable )

Attachments:

1. PM_amd64_8proc.png (57716 bytes)
2. 1.exe (548352 bytes)

Comment 4:

The amount of used physical memory does not seem to have changed in your graph. Can you
run 50 processes, as asked before, without problems?

Comment 5 by sjbogdan:

NO, program won't start without swap available
( program window doesn't show up, then Windows suggests to close other programs )

Attachments:

1. Untitled.png (9736 bytes)

Comment 6:

is your windows system set to disable swap (page file)?

Comment 7:

note:
64-bit Go program do allocate about 0.5 GB memory for it's heap metadata on
startup. But as the helloworld program shouldn't touch most of them, they
won't be backed by physical memory normally.
however, things might change if you disable swap (page file) completely.

Comment 8 by sjbogdan:

> is your windows system set to disable swap (page file)?
No
Can you run & show "Process Explorer"s memory info of my executable ?

Comment 9 by sjbogdan:

> 64-bit Go program do allocate about 0.5 GB memory for it's heap metadata on startup. >
But as the helloworld program shouldn't touch most of them, they won't be backed by
physical memory normally.
where can I read about it ?
it there any way to influence this ? ( build flags / system variables / other settings ?
)

Comment 10:

I linked to issue #5236. Did you read it at all?

Comment 11:

the relevant code is here:
https://golang.org/cl/7307122/diff/12002/src/pkg/runtime/malloc.goc#newcode316
my memory was wrong, the figure should be about 256 MB on amd64, and i just verified
that the mheap structure is 268480448 bytes on my system.
sorry, i don't have a 64-bit windows VM available to run arbitrary binaries. could you
please
verify that the same applies to godoc.exe in go1.1rc1 binary distribution?
you could run "godoc.exe -http localhost:6060" and then view its memory usage.

Comment 12 by sjbogdan:

> I linked to issue #5236. Did you read it at all?
Yes
- swap is not disabled ( a lot of talks and theory, not a single phrase about high
virtual memory usage )
- there is no patch for Windows amd64
- I'm not a linux system programmer, so can't tell whether it relates to Windows directly
- why there is such a huge difference between 8g vs 6g on Windows ?

Comment 13 by sjbogdan:

godoc

Attachments:

1. PM_amd64_godoc.png (19871 bytes)
2. PM1_amd64_godoc.png (9983 bytes)

Comment 14:

ok, i reproduced this using sysinternal's process explorer.
this is working as intended at least for now.
go 1.0.3 use 70MB committed memory at startup, go 1.1rc1 will use ~500MB.
go 1.0.3 could use at most 16GB heap whereas go 1.1rc1 supports 128GB,
so the committed memory usage is proportionate to maximum heap size.
this has something to do with windows memory management (it will reserve
page file even for memory that don't actually used.)
yes, this does mean that you can't run a large number of helloworld Go
programs on 64-bit windows without allocating a very large page file.
i will accept this issue, but i doubt we have simple solutions to this.
(we can lazily reserve the heap metadata, but that surely won't happen
before Go 1.1, this is issue #5236)
We need to investigate why the binary uses ~500MB of committed memory
instead of ~256MB as needed on unix.
after that, we can merge this issue with issue #5236.

Labels changed: added priority-later, expertneeded, removed priority-triage.

Status changed to Accepted.

Comment 16:

the issue tracker ate my reply two times.
i instrumented all calls to VirtualAlloc in pkg/runtime/mem_windows.c, and running
helloworld
program produces this trace:
<alloc 131072 bytes>
<alloc 268480448 bytes>
<reserve 146028888064 bytes>
<alloc 131072 bytes>
<map 1048576 bytes>
<map 65536 bytes>
<alloc 131072 bytes>
<alloc 65536 bytes>
<alloc 131072 bytes>
<alloc 4080 bytes>
<alloc 1048576 bytes>
This still cannot explain the 540MB of committed memory usage.

Comment 17 by sjbogdan:

Probably 540MB is related to <reserve 146028888064 bytes>, as it is a multiple of
256

Comment 18:

>This still cannot explain the 540MB of committed memory usage.
AFAIR, when you *reserve* memory on Windows, it *commits* memory for internal page
tables for that memory. This means that you can not reserve 100TB of virtual memory on
most Windows systems (as opposed to linux where it's fine).
If you want to conserve swap space on windows, you must not even reserve excessive
amounts of memory.

Comment 19:

Re #18, wow, neven though windows would do that, terrible.
Thank you for the explanation.
btw, so the race detector will use much more (committed) memory
on windows than on darwin/linux?

Labels changed: removed expertneeded.

Status changed to Duplicate.

Merged into issue #5236.

Comment 20:

> btw, so the race detector will use much more (committed) memory
on windows than on darwin/linux?
Because of that I had to switch to completely lazy allocation of memory in the race
detector. It does not allocate lots of memory ahead of time now. So it should be
comparable on darwin/linux/windows.

Comment 21:

I used vmmap.exe to check what is happening here. And most of allocated memory consist
of runtime·mheap (about 270MB) and "page table" (about 270MB). "Page table" is not
mapped into process virtual space, but it seems to be counted against process
"committed" memory. It is appears in process memory map as we "reserve" 128GB allocation
arena. From what I can gather, it is OS overhead of our reservation - perhaps some
tables that are used to translate process addresses into physical memory. The overhead
is present on 386 too, but, since reservation is much smaller, it is not as noticeable.
Even if we could avoid "page table" altogether, remaining 270MB is still too much for
average windows program.
Alex