code review 9245043: runtime: reduce max arena size on windows/amd64 to 32 GiB

runtime: reduce max arena size on windows/amd64 to 32 GiB
Update issue 5236
Update issue 5402
This CL reduces gofmt's committed memory from 545864 KiB to 139568 KiB.
Note: Go 1.0.3 uses about 70MiB.

[minux1, 2013-05-06 22:19:11]

Hello golang-dev@googlegroups.com (cc: golang-dev@googlegroups.com),

I'd like you to review this change to
https://code.google.com/p/go/

[minux1, 2013-05-06 22:21:24]

for the record,
the Go 1.0.3 max arena size is 16GiB.

quote from doc/go1.1.html:
"On 64-bit architectures, the maximum heap size has been enlarged substantially,
from a few gigabytes to several tens of gigabytes. (The exact details depend on
the system and may change.)"

changing max arena size to 32 GiB still fits the "several tens of gigabytes"
description.

[r, 2013-05-06 22:22:37]

Does this reduce the maximum size of the heap on Windows?

[minux1, 2013-05-06 22:23:05]

see
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx
for maximum supported memory of windows.

32 GiB seems reasonable.

[minux1, 2013-05-06 22:23:39]

On Tue, May 7, 2013 at 6:22 AM, <r@golang.org> wrote:

> Does this reduce the maximum size of the heap on Windows?
>
yes, from 128 GiB to 32 GiB.

[iant, 2013-05-06 22:40:28]

LGTM

Thanks.

https://codereview.appspot.com/9245043/diff/5001/src/pkg/runtime/malloc.h
File src/pkg/runtime/malloc.h (right):

https://codereview.appspot.com/9245043/diff/5001/src/pkg/runtime/malloc.h#new...
src/pkg/runtime/malloc.h:122: // Windows count memory used by page table into
committed memory
s/count/counts/

[r, 2013-05-06 22:41:30]

LGTM

[ioe, 2013-05-06 22:43:20]

small comment nit.

https://codereview.appspot.com/9245043/diff/5001/src/pkg/runtime/malloc.h
File src/pkg/runtime/malloc.h (right):

https://codereview.appspot.com/9245043/diff/5001/src/pkg/runtime/malloc.h#new...
src/pkg/runtime/malloc.h:118: // On 64-bit, we limit the arena to 128GB, or 37
bits.
On 64-bit, we limit the arena to 128GB, or 37 bits.
On Windows 64-bit we limit the arena to 32GB or 35 bits.

[minux1, 2013-05-06 22:53:21]

*** Submitted as https://code.google.com/p/go/source/detail?r=6f1ce1c15224 ***

runtime: reduce max arena size on windows/amd64 to 32 GiB
Update issue 5236
Update issue 5402
This CL reduces gofmt's committed memory from 545864 KiB to 139568 KiB.
Note: Go 1.0.3 uses about 70MiB.

R=golang-dev, r, iant, nightlyone
CC=golang-dev
https://codereview.appspot.com/9245043

https://codereview.appspot.com/9245043/diff/5001/src/pkg/runtime/malloc.h
File src/pkg/runtime/malloc.h (right):

https://codereview.appspot.com/9245043/diff/5001/src/pkg/runtime/malloc.h#new...
src/pkg/runtime/malloc.h:118: // On 64-bit, we limit the arena to 128GB, or 37
bits.
On 2013/05/06 22:43:20, ioe wrote:
> On 64-bit, we limit the arena to 128GB, or 37 bits.
> On Windows 64-bit we limit the arena to 32GB or 35 bits.
Thank you. Done.

https://codereview.appspot.com/9245043/diff/5001/src/pkg/runtime/malloc.h#new...
src/pkg/runtime/malloc.h:122: // Windows count memory used by page table into
committed memory
On 2013/05/06 22:40:28, iant wrote:
> s/count/counts/
Fixed.

[brainman, 2013-05-07 01:58:38]

LGTM. Thank you.

I don't use 64-bit windows myself, so I didn't notice the problem. But I do
remember increasing swap file size on our windows/amd64 builder twice recently.
I couldn't find the reason for the failure then. I know now.

Also, I suspect, having large blocks of committed memory that is not read or
written to doesn't affect performance as such, but it would fail new memory
allocations (OS wide) once limits (physical memory + swap size) are reached.

Even after this change, memory usage is too high. This program

package main

import "time"

func main () {
	time.Sleep ( 1 * time.Minute )
}

uses about 150MB of memory. I compared it with other windows programs on my
system - they are 20MB - 50MB. Still it is better then before.

Alex

[minux1, 2013-05-07 02:11:43]

On Tue, May 7, 2013 at 9:58 AM, <alex.brainman@gmail.com> wrote:

> Also, I suspect, having large blocks of committed memory that is not
> read or written to doesn't affect performance as such, but it would fail
> new memory allocations (OS wide) once limits (physical memory + swap
> size) are reached.
>
right. that's why we fix this before Go 1.1.
As explained in another thread, for a system with 4 GB memory (physical +
swap), before this change, one cannot start more than 8 Go program at
the same time, no matter how small the program is, which is ridiculous as
the system have plenty of physical RAM to actually run those programs.

>
> Even after this change, memory usage is too high. This program
>
> package main
>
> import "time"
>
> func main () {
>         time.Sleep ( 1 * time.Minute )
> }
>
> uses about 150MB of memory. I compared it with other windows programs on
> my system - they are 20MB - 50MB. Still it is better then before.
>
it's even suggested on the issue tracker that we reduce the limit to 8GiB or
even less, i'm fine with that but that will contradict our Go 1.1 release
note
(which states that heap size is tens of gigabytes).

[r, 2013-05-07 02:17:19]

The stopgap will do for now. In the longer term we need a better
answer that doesn't involve tuning an arbitrary constant.

-rob

[brainman, 2013-05-07 02:25:17]

On 2013/05/07 02:17:19, r wrote:
> The stopgap will do for now. In the longer term we need a better
> answer that doesn't involve tuning an arbitrary constant.
> 

My sentiment exactly.

Alex