Comment 1:

Labels changed: added priority-later, removed priority-medium.

Comment 2:

Issue #2177 has been merged into this issue.

Comment 4:

Issue #2385 has been merged into this issue.

Comment 5:

http://groups.google.com/group/golang-nuts/t/6f315a19b98254cb

Comment 6:

At some point I'm considering porting this project to go, but this issue is a blocker
for me. Ideally I'd like to be able to use as much RAM as a server has available,
typically 32 or 64GB in our current use cases.
https://github.com/mediastandardstrust/superfastmatch

Comment 7:

You can already do allocation outside of the size-limited Go heap by writing a bit of
stuff on top of a raw mmap syscall that uses uintptrs instead of syscall.Mmap's []byte
interface. You would then expose your big buffer as many smaller 2 GB []bytes. Not
ideal, but manageable. With the current GC (or any GC) you probably don't want many
gigabytes of small objects anyway.
The same can probably be done with Alloc (if that's the right name) on Windows.

Comment 8:

Hi,
thanks for the suggestion and leads! I'm currently using sparsetable from the sparsehash
project in the C++ version to manage up to 1<<32 pointers to variable length
blocks. The overhead per block is 2.36 bits plus 8 bytes for the pointer itself on
64-bit. It's effectively a huge hash table where the hashed value itself is discarded
with sparsetable being abused as a memory allocation manager! The hash table is slotted
by ranges to allow multi-threaded access. Obviously there is the overhead of the malloc
implementation as well (tried tcmalloc, faster-but eats memory!). 
Your suggestion would involve writing a custom memory allocator which does sound like a
more difficult proposition :) I wonder if there's any internal Google projects to do
with threadsafe/goroutinesafe sparse hash tables in Go? I suppose the requirement is
some kind of bitmapped memory allocation with GC. All memory allocators need to
deallocate memory at some point anyway, so the question is who is best to do it :)
Cheers,
Donovan.

Comment 9:

If you really need to be able to use TBs of memory, then there seems to be a temporary
fix as suggested by Russ (
https://groups.google.com/d/msg/golang-nuts/bzFaGbmCVMs/Fn8Fdv3-KIEJ ). Change the
memory limit by editing src/pkg/runtime/malloc.goc and changing '16LL<<30' to
something larger. (But this may cause a segv with gob.Encode or some other unknown
problems).

Comment 10:

If you have a look at issue #2177, you'll see that Russ' suggestion does not work
unfortunately - even in the simplest of cases.

Comment 11:

Further experimentation on this.
This change allows allocation and addressing of data beyond 16GB (in this case 64GB):
diff -r 920e9d1ffd1f src/pkg/runtime/malloc.goc
--- a/src/pkg/runtime/malloc.goc    Wed Mar 28 23:41:59 2012 +1100
+++ b/src/pkg/runtime/malloc.goc    Sun Jun 10 09:39:58 2012 +0930
@@ -305,7 +305,7 @@
        // allocate 15 GB before we get that far.
        //
        // If this fails we fall back to the 32 bit memory mechanism
-       arena_size = 16LL<<30;
+       arena_size = 16LL<<32;
        bitmap_size = arena_size / (sizeof(void*)*8/4);
        p = runtime·SysReserve((void*)(0x00f8ULL<<32), bitmap_size + arena_size);
    }
diff -r 920e9d1ffd1f src/pkg/runtime/malloc.h
--- a/src/pkg/runtime/malloc.h  Wed Mar 28 23:41:59 2012 +1100
+++ b/src/pkg/runtime/malloc.h  Sun Jun 10 09:39:58 2012 +0930
@@ -116,7 +116,7 @@
    // On 64-bit, we limit the arena to 16G, so 22 bits suffices.
    // On 32-bit, we don't bother limiting anything: 20 bits for 4G.
 #ifdef _64BIT
-   MHeapMap_Bits = 22,
+   MHeapMap_Bits = 24,
 #else
    MHeapMap_Bits = 20,
 #endif
It may well have bad interactions with garbage collection, but could be a work around in
the short term if that is not an issue.

Comment 12:

The change Dan suggested seems to work fine - GC was slow, but still released memory
when over 100GB of RAM was used. I used basic primitives - structs, interfaces, arrays,
maps, channels.

Comment 13:

Labels changed: added go1.1.

Comment 14:

Very nice, thank you for identifying that fix.

Comment 15:

This may be useful, http://golang.org/cl/6343086.

Comment 16:

Here's a story from the trenches that could be considered.
We have been running Go services under systemd on Linux. We have also been using the
systemd support for cgroups to limit the memory these processes can allocate.
There was some discussion on systemd-devel:
http://lists.freedesktop.org/archives/systemd-devel/2012-March/004802.html
http://lists.freedesktop.org/archives/systemd-devel/2012-March/004811.html
http://lists.freedesktop.org/archives/systemd-devel/2012-March/004826.html
http://lists.freedesktop.org/archives/systemd-devel/2012-March/004810.html
http://lists.freedesktop.org/archives/systemd-devel/2012-March/004823.html
http://www.kernel.org/doc/Documentation/cgroups/memory.txt
As far as we understand memory.txt, cgroups memory limits accounts mapped files under
some conditions, so it doesn't work for limiting a Go process that should have a small
heap but be able to map terabytes of files. As far as I can tell, RSS limits aren't
implemented on Linux.
We see the following kinds of programs:
1. Go programs that don't map files
2. Go programs that call cgo libraries (that might leak)
3. Go programs that map files
4. Go programs that call cgo libraries and map files
Some programs must simply never hit their memory limits under normal operation. Some
programs that interact with connections from many other programs must potentially be
able to throttle or later reject work depending on how close to their memory limit they
are.
For pure Go programs that don't map files, you can use cgroups. You can even use both
the hard memory limit and a soft memory limit and using an eventfd (see memory.txt) or
reading your limits you can throttle back or cancel actions in your service to limit its
memory. Currently you need to keep both these limits a bit below 16 GB to avoid hitting
the heap limit.
Any kind of program that maps files is tricky. You can't use cgroups for these kinds of
programs in general. It can be useful to limit the Go heap, but depending on the
service, you might want to do different things when you go over the soft limit. You
don't necessarily want an infinite hard limit. Eventually the kernel's OOM killer might
kill you, but it might kill someone else first.
Go+leaking cgo+mapped files is the hardest. The best you can do here is probably to fix
the leaks and then just limit the Go part.
Finally, this leads me to my proposal. Once the 16 GB limit is lifted (which allows apps
to potentially grow huge heaps), it might be useful to be able to instruct the runtime
to still limit the heap to cater for cases where cgroups can't help you.
It's probably a bit sucky to do it via environment variables or flags understood by all
Go programs (kind of like the JVM -Xmx stuff), so people that want it can do their own
flags and call runtime functions.
I think a soft limit in the runtime is also useful (maybe with some kind of channel API,
like you can get with the cgroups eventfd thing) as it allows goroutines handling
transactions to temporarily pause work (maybe inside a select that combines a limit
checker, ticker, and other channels) when you go over the soft limit, but before the
runtime starts failing allocations due to hitting the hard limit.
P.S. the cgroups code for reading limits looks something like this. We haven't done
anything with the eventfds yet. This code assumes that the limits don't change while the
program is running.
func MemoryLimit() int64 {
    return memoryLimit
}
func MemorySoftLimit() int64 {
    return memorySoftLimit
}
func MemoryUsage() int64 {
    usageStr := readLine(memoryUsagePath)
    n, err := strconv.ParseUint(usageStr, 0, 64)
    if err != nil {
        return 0
    }
    return int64(n)
}
var memoryLimit int64
var memorySoftLimit int64
var memoryUsagePath string
func init() {
    var path string
    lines := readLines("/proc/self/cgroup")
    for _, line := range lines {
        if !strings.Contains(line, ":memory:") {
            continue
        }
        parts := strings.Split(line, ":")
        if len(parts) != 3 {
            continue
        }
        path = "/sys/fs/cgroup/memory" + parts[2]
    }
    if len(path) == 0 {
        return
    }
    softLimitStr := readLine(path + "/memory.soft_limit_in_bytes")
    n, err := strconv.ParseUint(softLimitStr, 0, 64)
    if err == nil {
        memorySoftLimit = int64(n)
    }
    limitStr := readLine(path + "/memory.limit_in_bytes")
    n, err = strconv.ParseUint(limitStr, 0, 64)
    if err == nil {
        memoryLimit = int64(n)
    }
    memoryUsagePath = path + "/memory.usage_in_bytes"
}

Comment 17:

Note that CL 6551067 introduces a MaxMem constant in malloc.h that bounds the size of
any single allocation via makeslice or makechan on 64-bit machines to 16 GB. If you
expand this amount, that constant needs expansion too.

Comment 18:

One constraint about the address choice is that I want to make sure that heap pointers
never appear in valid UTF-8 sequences, for the imprecise garbage collection of at least
stacks if not the whole heap. The current pointers are in the range 0xf8<<32 up to
(but not including) 0xfc<<32, and f8, f9, fa, fb never appear in UTF-8 sequences.
We can expand the current space to ff, so 32 GB, before hitting valid UTF-8.
An alternative would be to use 0x0080<<32 up to (but not including)
0x00C0<<32, because 80-bf never appear in UTF-8 following an ASCII byte like 00.
That would be 256 GB worth of address space.

Comment 19:

Actually the bytes are backward because this is little-endian. So we should use
0x00C0<<32 because 0xC0 and later are never followed by a 00 byte.

Comment 20:

http://golang.org/cl/6826088

Comment 21:

This issue was closed by revision 9799a5a.

Status changed to Fixed.