Fuller trace:

runtime: memory allocated by OS (1626533888) not in usable range [0x9143a000,0x1143a000)
runtime: out of memory: cannot allocate 268435456-byte block (3145728 in use)
fatal error: out of memory
runtime stack:
 runtime.throw(0xa23d7a60, 0xd)
    /usr/local/go/src/runtime/panic.go:530 +0x80
 runtime.largeAlloc(0x10000000, 0x1, 0x25)
    /usr/local/go/src/runtime/malloc.go:764 +0xe8
runtime.mallocgc.func3()
    /usr/local/go/src/runtime/malloc.go:660 +0x34
runtime.systemstack(0xbed29de4)
    /usr/local/go/src/runtime/asm_arm.s:243 +0x8c
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1042
goroutine 17 [running, locked to thread]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_arm.s:188 +0x4 fp=0x91476cb8 sp=0x91476cb4
runtime.mallocgc(0x10000000, 0xa221c4c0, 0x1, 0x4)
    /usr/local/go/src/runtime/malloc.go:661 +0xb04 fp=0x91476d28 sp=0x91476cb8
runtime.newarray(0xa221c4c0, 0x4000000, 0x91658000)
    /usr/local/go/src/runtime/malloc.go:794 +0xec fp=0x91476d48 sp=0x91476d28
runtime.makeslice(0xa2210118, 0x4000000, 0x0, 0x4000000, 0x0, 0x0, 0x0, 0x0)

This can happen on a 32-bit system if your program is using too much memory--that is, if it is going to run out of memory anyhow. One thing to check is whether the program has a bug causing it to use memory without bounds.

It is also possible that this is a Go runtime error. Does this error happen consistently? Is there a way that we can reproduce the problem?

I'll try to create a repro of this issue.

It says you are allocating a 256 MB slice. Is that true?

On a 32-bit system, especially one co-existing with C code as in Android, there's just not that much address space to go around. The Android L runtime ART reserves its own virtual memory quite aggressively, so Go often can't get much more. What happened here is that Go asked for 256 MB for its heap bitmap (mostly that will never be mapped, it's just virtual address space), 2 MB for its span array, and some amount for its own memory. That amount is unclear: we try 512 MB, and if that fails then 256 MB, and if that fails then 128 MB. It's hard to say what finally worked here, but the kernel found the memory we wanted and gave it to us at 0x8123a000 or so, resulting in a heap base of 0x9143a000. Then when we asked for the 256 MB later, the OS gave us 0x60f2f000 (1626533888), which was below the heap base and therefore unusable to us. We need it to be in the 2 GB starting at the original arena base to be able to address it in our heap bitmap.

The code that goes through this in runtime/malloc.go says:

    // On a 32-bit machine, we can't typically get away
    // with a giant virtual address space reservation.
    // Instead we map the memory information bitmap
    // immediately after the data segment, large enough
    // to handle another 2GB of mappings (256 MB),
    // along with a reservation for an initial arena.
    // When that gets used up, we'll start asking the kernel
    // for any memory anywhere and hope it's in the 2GB
    // following the bitmap (presumably the executable begins
    // near the bottom of memory, so we'll have to use up
    // most of memory before the kernel resorts to giving out
    // memory before the beginning of the text segment).
    //
    // Alternatively we could reserve 512 MB bitmap, enough
    // for 4GB of mappings, and then accept any memory the
    // kernel threw at us, but normally that's a waste of 512 MB
    // of address space, which is probably too much in a 32-bit world.

    // If we fail to allocate, try again with a smaller arena.
    // This is necessary on Android L where we share a process
    // with ART, which reserves virtual memory aggressively.
    arenaSizes := []uintptr{
        512 << 20,
        256 << 20,
        128 << 20,
    }

    for _, arenaSize := range arenaSizes {
        bitmapSize = _MaxArena32 / (sys.PtrSize * 8 / 4)
        spansSize = _MaxArena32 / _PageSize * sys.PtrSize

The comment is somewhat out of date. In fact the code is reserving 256 MB of heap bitmap when it only needs to reserve 128 MB. We halved the footprint without updating the calculation of bitmapSize on that next-to-last line. It's too late for Go 1.6 but we could do one of two things in Go 1.7:

(1) Halve the bitmapSize, since that's all we need for _MaxArena32 = 2GB.

(2) Double _MaxArena32. That's what the "Alternatively ..." comment above is suggesting, but there the assumption is it will require 512 MB of bitmap. Now that bitmaps are half as big, we can keep using the same 256 MB we're already reserving and use it to describe the entire 4 GB space, letting us eliminate this class of crash entirely.

I suspect we should do (2). But again it will need to wait for Go 1.7.

/cc @RLH @aclements

CL https://golang.org/cl/18975 mentions this issue.

We've found a bug in our code that used more memory than we anticipated it would, that may have caused running out of memory, and hitting some weird limits with the address space on 32 bit systems. If you wish you can close down this thread. I'll open a new one or post into this if the issue resurfaces.

Thanks. Please reopen if the situation changes.

@davecheney, are you sure? I think Russ' suggestions sounds good to me, eliminating this kind of crash entirely.

It's the OP's choice, see above.

I only said that we found a bug that increased the memory usage :P If there's a fix that would make the code more stable on 32 bit platforms I'm all for it :)

CL https://golang.org/cl/20370 mentions this issue.

If you wish you can close down this thread. I'll open a new one or post into this if the issue resurfaces.

Ok, my mistake. Reopening.

CL https://golang.org/cl/20471 mentions this issue.

I think the fix should be pushed to 1.6 branch. Building docker with go 1.6.2 fails on arm but works with 1.6.1.

@ncopa, the fix for this issue is quite invasive and it's fixing something that has been a problem for many Go releases (that is, the problem wasn't introduced in Go 1.6; it certainly wasn't introduced in Go 1.6.2), so I'm afraid it isn't really a candidate for cherry-picking to 1.6.

I'm surprised Docker built on ARM with 1.6.1 but not with 1.6.2. You could open a new issue about that with more details.

@aclements yes. I should have opened a new issue. I tried to backport the fix but it didnt solve the it. Must be something else that is happening.