Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: 512GB memory limitation #10460

Closed
ndaniels opened this issue Apr 14, 2015 · 23 comments
Closed

runtime: 512GB memory limitation #10460

ndaniels opened this issue Apr 14, 2015 · 23 comments
Milestone

Comments

@ndaniels
Copy link

In Go 1.4, the compiler still has a hardcoded 128GB memory limit. While this is much better than the 16GB limit from 1.0, it's presenting a real limitation to using Go for big-data scientific work, such as my research, such as https://github.com/ndaniels/cablastp2. While it's trivial for me to change the allocation limit in go's source and rebuild my own compiler, this presents a problem for other users using my work; if they find out that they need to patch the Go compiler just to build my source code, they're not going to like it. It's already enough of a barrier for some users to require they have the Go compiler. While I could provide statically compiled binaries for a variety of platforms, I can't target every OS and kernel version, and it becomes a nuisance to update those binaries whenever I update my source code.

I understand Go's memory allocation philosophy, and I realize that (for the time being) it needs to be hardcoded in a Go binary. But, it would be a huge improvement if a compiler flag could be used to alter the allocation limit for a particular package being compiled. Even better would be a runtime flag (much like Java's Xmx flag for the heap).

@ianlancetaylor ianlancetaylor changed the title 128GB memory limit - could this be a compiler flag? runtime: 128GB memory limit - could this be a compiler flag? Apr 14, 2015
@ianlancetaylor
Copy link
Contributor

To be clear, I think you are talking about the value of MaxMem in runtime/malloc.go (tip) or runtime/malloc.h (Go 1.4), right?

@ianlancetaylor ianlancetaylor added this to the Go1.5Maybe milestone Apr 14, 2015
@ndaniels
Copy link
Author

Yes, exactly. Couldn't that be a default, but have it changeable via a compiler flag? I mean when calling 6g, not when compiling 6g.

@minux
Copy link
Member

minux commented Apr 15, 2015 via email

@ndaniels
Copy link
Author

I think an explicit command-line argument to the binary (handled by the runtime, perhaps; the GHC Haskell compiler, for example, does this with -rtsopts) is more intuitive and discoverable, and less likely to lead to surprising effects. That would be ideal; a reasonable fallback would be a compile-time flag. Certainly it's better than requiring users to patch the compiler source code!

@minux
Copy link
Member

minux commented Apr 15, 2015 via email

@ndaniels
Copy link
Author

I certainly understand that something like a "-goruntime" flag can't be added until Go 2.

I think there is definite interest in Go in data science, and the need for more than 128GB is going to be common enough. So, perhaps adding an environment variable is warranted.

How does one change MaxMem at runtime (of the compiler, I presume)? Up until 1.4 it certainly required a source change to malloc.(go,h); I recall patching it for 1.0 when 16GB didn't cut it.

@bradfitz bradfitz changed the title runtime: 128GB memory limit - could this be a compiler flag? runtime: 128GB memory limitation Apr 15, 2015
@RLH
Copy link
Contributor

RLH commented Apr 15, 2015

If we want to increase this to say a Terabyte I don't see why a Go flag or
environment variable is needed, reflecting the limits of the underlying
OS/HW seems the correct approach. I don't have a sense of what, in terms
of allocation rate, pointer density, sharing between Goroutines, and
pointer mutation, would fill a Terabyte of Go memory.

On Tue, Apr 14, 2015 at 11:07 PM, Noah Daniels notifications@github.com
wrote:

I certainly understand that something like a "-goruntime" flag can't be
added until Go 2.

I think there is definite interest in Go in data science, and the need for
more than 128GB is going to be common enough. So, perhaps adding an
environment variable is warranted.

How does one change MaxMem at runtime (of the compiler, I presume)? Up
until 1.4 it certainly required a source change to malloc.(go,h); I recall
patching it for 1.0 when 16GB didn't cut it.


Reply to this email directly or view it on GitHub
#10460 (comment).

@rsc
Copy link
Contributor

rsc commented Jun 8, 2015

It is not trivial to just change MaxMem at run time, and it's probably too late in the cycle to introduce code allowing MaxMem to be a variable, although I agree it is probably the right long-term solution.

I will, however, bump MaxMem on 64-bit systems to 512 GB for Go 1.5. That should buy us some more headroom until we can fix things properly.

@gopherbot
Copy link

CL https://golang.org/cl/10819 mentions this issue.

@ndaniels
Copy link
Author

"I don't have a sense of what, in terms
of allocation rate, pointer density, sharing between Goroutines, and
pointer mutation, would fill a Terabyte of Go memory."

The National Center for Biotechnology Information (NCBI) maintains a database of "non-redundant" (unique) protein sequences. That database is now over 60 million sequences, and for the past several years has been doubling every two years.

PubChem (a public database of chemical compounds) is over 300GB in size on disk. It has also been growing, though not as quickly.

If Go is going to be useful for big-data science, it needs to be able to use as much RAM as we can throw into a system. Terabyte-plus compute servers are common now. Now, I don't need a terabyte today, but I need more than 128GB. The right solution is for this not to be hardcoded.

In my case, what consumes all the RAM is a large lookup table of kmers (very short sequences) indexed into every sequence in the database I'm trying to compress. Lots of pointers, lots of string data, and we don't want to write things out to disk until we're done.

@davecheney
Copy link
Contributor

@ndaniels understood. This is not a solution, just a stop gap, see the comment by @rsc from three days ago.

rsc added a commit that referenced this issue Jun 15, 2015
A workaround for #10460.

Change-Id: I607a556561d509db6de047892f886fb565513895
Reviewed-on: https://go-review.googlesource.com/10819
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
@rsc rsc modified the milestones: Go1.6, Go1.5Maybe Jun 17, 2015
@rsc rsc changed the title runtime: 128GB memory limitation runtime: 512GB memory limitation Jun 17, 2015
@rsc rsc modified the milestones: Unplanned, Go1.6 Nov 5, 2015
@pbnjay
Copy link
Contributor

pbnjay commented Jun 30, 2017

I wanted to raise this issue again, I know it's been a while but I've hit the 512GB limit on a system with 4TB. It took me a while to determine that Go's runtime was the issue, and not something else. Ideally, the out of memory error message could report that this is an internal Go limit reached and not what looks like a system OOM message.

I feel 512GB is perfectly reasonable as a default, but it would be great to document and provide a way for make.bash et al to update this limit. I've simply hardcoded _MHeapMap_TotalBits = 42 for now but it feels brittle.

FWIW, I'm also doing bioinformatics work like @ndaniels - Go is very well suited to it.

@rsc rsc modified the milestones: Go1.10, Unplanned Jul 6, 2017
@rsc
Copy link
Contributor

rsc commented Jul 6, 2017

/cc @aclements to think about for Go 1.10.

@rajakhurram
Copy link

+1 @pbnjay
At the minimum, Golang should exit with a meaningful panic() when the system hits 512GB limit. This should really be released as a fix in 1.8 and should not wait for the next major release.
We are using memory mapped files in our system, and this means that our virtual address space can be huge - virtually unlimited. Please either provide a compile time flag for setting the limit or automatically extend it at runtime to honor "ulimit -v = unlimited" on linux systems.

@aclements
Copy link
Member

At the minimum, Golang should exit with a meaningful panic() when the system hits 512GB limit.

This would require threading some error information around that we currently don't, but should be pretty easy.

We are using memory mapped files in our system, and this means that our virtual address space can be huge - virtually unlimited.

I'm not sure what problem you're getting at here. Memory mapped files have no effect on Go's heap arena limit since they don't live in the Go heap.

Please either provide a compile time flag for setting the limit or automatically extend it at runtime to honor "ulimit -v = unlimited" on linux systems.

I would love to eliminate this limit, but doing so requires a complete redesign of the Go heap. I've actually been thinking about how to do that, but it's not a simple matter of just extending it at runtime. :)

@aclements aclements modified the milestones: Go1.10, Go1.11 Nov 8, 2017
@gopherbot
Copy link

Change https://golang.org/cl/85887 mentions this issue: runtime: use sparse mappings for the heap

@gopherbot
Copy link

Change https://golang.org/cl/85889 mentions this issue: runtime: move comment about address space sizes to malloc.go

@gopherbot
Copy link

Change https://golang.org/cl/85885 mentions this issue: runtime: make span map sparse

@gopherbot
Copy link

Change https://golang.org/cl/85888 mentions this issue: runtime: remove non-reserved heap logic

@gopherbot
Copy link

Change https://golang.org/cl/85884 mentions this issue: runtime: abstract remaining mheap.spans access

@gopherbot
Copy link

Change https://golang.org/cl/85882 mentions this issue: runtime: fix various contiguous bitmap assumptions

@gopherbot
Copy link

Change https://golang.org/cl/85883 mentions this issue: runtime: make the heap bitmap sparse

@gopherbot
Copy link

Change https://golang.org/cl/85886 mentions this issue: runtime: eliminate most uses of mheap_.arena_*

gopherbot pushed a commit that referenced this issue Feb 15, 2018
There are various places that assume the heap bitmap is contiguous and
scan it sequentially. We're about to split up the heap bitmap. This
commit modifies all of these except heapBitsSetType to use the
heapBits abstractions so they can transparently switch to a
discontiguous bitmap.

Updates #10460. This is a step toward supporting sparse heaps.

Change-Id: I2f3994a5785e4dccb66602fb3950bbd290d9392c
Reviewed-on: https://go-review.googlesource.com/85882
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>
gopherbot pushed a commit that referenced this issue Feb 15, 2018
This splits the heap bitmap into separate chunks for every 64MB of the
heap and introduces an index mapping from virtual address to metadata.
It modifies the heapBits abstraction to use this two-level structure.
Finally, it modifies heapBitsSetType to unroll the bitmap into the
object itself and then copy it out if the bitmap would span
discontiguous bitmap chunks.

This is a step toward supporting general sparse heaps, which will
eliminate address space conflict failures as well as the limit on the
heap size.

It's also advantageous for 32-bit. 32-bit already supports
discontiguous heaps by always starting the arena at address 0.
However, as a result, with a contiguous bitmap, if the kernel chooses
a high address (near 2GB) for a heap mapping, the runtime is forced to
map up to 128MB of heap bitmap. Now the runtime can map sections of
the bitmap for just the parts of the address space used by the heap.

Updates #10460.

This slightly slows down the x/garbage and compilebench benchmarks.
However, I think the slowdown is acceptably small.

name        old time/op     new time/op     delta
Template        178ms ± 1%      180ms ± 1%  +0.78%    (p=0.029 n=10+10)
Unicode        85.7ms ± 2%     86.5ms ± 2%    ~       (p=0.089 n=10+10)
GoTypes         594ms ± 0%      599ms ± 1%  +0.70%    (p=0.000 n=9+9)
Compiler        2.86s ± 0%      2.87s ± 0%  +0.40%    (p=0.001 n=9+9)
SSA             7.23s ± 2%      7.29s ± 2%  +0.94%    (p=0.029 n=10+10)
Flate           116ms ± 1%      117ms ± 1%  +0.99%    (p=0.000 n=9+9)
GoParser        146ms ± 1%      146ms ± 0%    ~       (p=0.193 n=10+7)
Reflect         399ms ± 0%      403ms ± 1%  +0.89%    (p=0.001 n=10+10)
Tar             173ms ± 1%      174ms ± 1%  +0.91%    (p=0.013 n=10+9)
XML             208ms ± 1%      210ms ± 1%  +0.93%    (p=0.000 n=10+10)
[Geo mean]      368ms           371ms       +0.79%

name                       old time/op  new time/op  delta
Garbage/benchmem-MB=64-12  2.17ms ± 1%  2.21ms ± 1%  +2.15%  (p=0.000 n=20+20)

Change-Id: I037fd283221976f4f61249119d6b97b100bcbc66
Reviewed-on: https://go-review.googlesource.com/85883
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>
gopherbot pushed a commit that referenced this issue Feb 15, 2018
This abstracts the remaining direct accesses to mheap.spans into new
mheap.setSpan and mheap.setSpans methods.

For #10460.

Change-Id: Id1db8bc5e34a77a9221032aa2e62d05322707364
Reviewed-on: https://go-review.googlesource.com/85884
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>
gopherbot pushed a commit that referenced this issue Feb 15, 2018
This splits the span map into separate chunks for every 64MB of the
heap. The span map chunks now live in the same indirect structure as
the bitmap.

Updates #10460.

This causes a slight improvement in compilebench and the x/benchmarks
garbage benchmark. I'm not sure why it improves performance.

name       old time/op     new time/op     delta
Template       185ms ± 1%      184ms ± 1%    ~            (p=0.315 n=9+10)
Unicode       86.9ms ± 1%     86.9ms ± 3%    ~            (p=0.356 n=9+10)
GoTypes        602ms ± 1%      599ms ± 0%  -0.59%         (p=0.002 n=9+10)
Compiler       2.89s ± 0%      2.87s ± 1%  -0.50%          (p=0.003 n=9+9)
SSA            7.25s ± 0%      7.29s ± 1%    ~            (p=0.400 n=9+10)
Flate          118ms ± 1%      118ms ± 2%    ~            (p=0.065 n=10+9)
GoParser       147ms ± 2%      147ms ± 1%    ~            (p=0.549 n=10+9)
Reflect        403ms ± 1%      401ms ± 1%  -0.47%         (p=0.035 n=9+10)
Tar            176ms ± 1%      175ms ± 1%  -0.59%         (p=0.013 n=10+9)
XML            211ms ± 1%      209ms ± 1%  -0.83%        (p=0.011 n=10+10)

(https://perf.golang.org/search?q=upload:20171231.1)

name                       old time/op  new time/op  delta
Garbage/benchmem-MB=64-12  2.24ms ± 1%  2.23ms ± 1%  -0.36%  (p=0.001 n=20+19)

(https://perf.golang.org/search?q=upload:20171231.2)

Change-Id: I2563f8704ab9812434947faf293c5327f9b0d07a
Reviewed-on: https://go-review.googlesource.com/85885
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>
gopherbot pushed a commit that referenced this issue Feb 15, 2018
This replaces all uses of the mheap_.arena_* fields outside of
mallocinit and sysAlloc. These fields fundamentally assume a
contiguous heap between two bounds, so eliminating these is necessary
for a sparse heap.

Many of these are replaced with checks for non-nil spans at the test
address (which in turn checks for a non-nil entry in the heap arena
array). Some of them are just for debugging and somewhat meaningless
with a sparse heap, so those we just delete.

Updates #10460.

Change-Id: I8345b95ffc610aed694f08f74633b3c63506a41f
Reviewed-on: https://go-review.googlesource.com/85886
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>
gopherbot pushed a commit that referenced this issue Feb 15, 2018
Currently large sysReserve calls on some OSes don't actually reserve
the memory, but just check that it can be reserved. This was important
when we called sysReserve to "reserve" many gigabytes for the heap up
front, but now that we map memory in small increments as we need it,
this complication is no longer necessary.

This has one curious side benefit: currently, on Linux, allocations
that are large enough to be rejected by mmap wind up freezing the
application for a long time before it panics. This happens because
sysReserve doesn't reserve the memory, so sysMap calls mmap_fixed,
which calls mmap, which fails because the mapping is too large.
However, mmap_fixed doesn't inspect *why* mmap fails, so it falls back
to probing every page in the desired region individually with mincore
before performing an (otherwise dangerous) MAP_FIXED mapping, which
will also fail. This takes a long time for a large region. Now this
logic is gone, so the mmap failure leads to an immediate panic.

Updates #10460.

Change-Id: I8efe88c611871cdb14f99fadd09db83e0161ca2e
Reviewed-on: https://go-review.googlesource.com/85888
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>
gopherbot pushed a commit that referenced this issue Feb 15, 2018
Currently there's a detailed comment in lfstack_64bit.go about address
space limitations on various architectures. Since that's now relevant
to malloc, move it to a more prominent place in the documentation for
memLimitBits.

Updates #10460.

Change-Id: If9708291cf3a288057b8b3ba0ba6a59e3602bbd6
Reviewed-on: https://go-review.googlesource.com/85889
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>
@golang golang locked and limited conversation to collaborators Feb 15, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

10 participants