Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: memory allocated by OS not in usable range (ARM) #13992

Closed
karalabe opened this issue Jan 18, 2016 · 17 comments
Closed

runtime: memory allocated by OS not in usable range (ARM) #13992

karalabe opened this issue Jan 18, 2016 · 17 comments
Milestone

Comments

@karalabe
Copy link
Contributor

Hey all,

I'm not sure whether this is a bug in Go, a bug in our code or something just odd, but thought I'd post it to get some pointers, as previous occurrences of this error usually meant bugs in the runtime.

One of our users were experimenting with running our Ethereum codebase on Android, ARMv7 with Go 1.6 beta1 or beta2 (didn't get a confirmation yet on it), and at a certain point he was hit with the non-usable memory error below:

runtime: memory allocated by OS (1626533888) not in usable range [0x9143a000,0x1143a000)
runtime: out of memory: cannot allocate 268435456-byte block (3145728 in use)

Could this point to an underlying issue in Go, or should I look in our own code base? Given the oddity of the report that memory seems to be available but Go cannot use it, it seems to me that it's an issue with Go, but I'm not sure of it.


Update: I think the Android library producing this issue was built with 1.6beta1 (via xgo, notable this .aar bundle), will ask for a repro code if it's deemed to be a Go bug requiring investigation. The code was ran on a Sony Xperia Z2, which should have plenty of memory to cover the code's requirements.

Update 2: Given that the cross builds are from xgo, both ARMv7 and ARM64 libraries are included in there, so the issue might be with ARM64 and not ARMv7 as I originally thought.

@karalabe karalabe changed the title runtime: memory allocated by OS not in usable range (armv7) runtime: memory allocated by OS not in usable range (ARM) Jan 18, 2016
@karalabe
Copy link
Contributor Author

Fuller trace:

runtime: memory allocated by OS (1626533888) not in usable range [0x9143a000,0x1143a000)
runtime: out of memory: cannot allocate 268435456-byte block (3145728 in use)
fatal error: out of memory
runtime stack:
 runtime.throw(0xa23d7a60, 0xd)
    /usr/local/go/src/runtime/panic.go:530 +0x80
 runtime.largeAlloc(0x10000000, 0x1, 0x25)
    /usr/local/go/src/runtime/malloc.go:764 +0xe8
runtime.mallocgc.func3()
    /usr/local/go/src/runtime/malloc.go:660 +0x34
runtime.systemstack(0xbed29de4)
    /usr/local/go/src/runtime/asm_arm.s:243 +0x8c
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1042
goroutine 17 [running, locked to thread]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_arm.s:188 +0x4 fp=0x91476cb8 sp=0x91476cb4
runtime.mallocgc(0x10000000, 0xa221c4c0, 0x1, 0x4)
    /usr/local/go/src/runtime/malloc.go:661 +0xb04 fp=0x91476d28 sp=0x91476cb8
runtime.newarray(0xa221c4c0, 0x4000000, 0x91658000)
    /usr/local/go/src/runtime/malloc.go:794 +0xec fp=0x91476d48 sp=0x91476d28
runtime.makeslice(0xa2210118, 0x4000000, 0x0, 0x4000000, 0x0, 0x0, 0x0, 0x0)

@ianlancetaylor ianlancetaylor added this to the Go1.6Maybe milestone Jan 18, 2016
@ianlancetaylor
Copy link
Contributor

This can happen on a 32-bit system if your program is using too much memory--that is, if it is going to run out of memory anyhow. One thing to check is whether the program has a bug causing it to use memory without bounds.

It is also possible that this is a Go runtime error. Does this error happen consistently? Is there a way that we can reproduce the problem?

@karalabe
Copy link
Contributor Author

I'll try to create a repro of this issue.

@rsc
Copy link
Contributor

rsc commented Jan 27, 2016

It says you are allocating a 256 MB slice. Is that true?

On a 32-bit system, especially one co-existing with C code as in Android, there's just not that much address space to go around. The Android L runtime ART reserves its own virtual memory quite aggressively, so Go often can't get much more. What happened here is that Go asked for 256 MB for its heap bitmap (mostly that will never be mapped, it's just virtual address space), 2 MB for its span array, and some amount for its own memory. That amount is unclear: we try 512 MB, and if that fails then 256 MB, and if that fails then 128 MB. It's hard to say what finally worked here, but the kernel found the memory we wanted and gave it to us at 0x8123a000 or so, resulting in a heap base of 0x9143a000. Then when we asked for the 256 MB later, the OS gave us 0x60f2f000 (1626533888), which was below the heap base and therefore unusable to us. We need it to be in the 2 GB starting at the original arena base to be able to address it in our heap bitmap.

The code that goes through this in runtime/malloc.go says:

    // On a 32-bit machine, we can't typically get away
    // with a giant virtual address space reservation.
    // Instead we map the memory information bitmap
    // immediately after the data segment, large enough
    // to handle another 2GB of mappings (256 MB),
    // along with a reservation for an initial arena.
    // When that gets used up, we'll start asking the kernel
    // for any memory anywhere and hope it's in the 2GB
    // following the bitmap (presumably the executable begins
    // near the bottom of memory, so we'll have to use up
    // most of memory before the kernel resorts to giving out
    // memory before the beginning of the text segment).
    //
    // Alternatively we could reserve 512 MB bitmap, enough
    // for 4GB of mappings, and then accept any memory the
    // kernel threw at us, but normally that's a waste of 512 MB
    // of address space, which is probably too much in a 32-bit world.

    // If we fail to allocate, try again with a smaller arena.
    // This is necessary on Android L where we share a process
    // with ART, which reserves virtual memory aggressively.
    arenaSizes := []uintptr{
        512 << 20,
        256 << 20,
        128 << 20,
    }

    for _, arenaSize := range arenaSizes {
        bitmapSize = _MaxArena32 / (sys.PtrSize * 8 / 4)
        spansSize = _MaxArena32 / _PageSize * sys.PtrSize

The comment is somewhat out of date. In fact the code is reserving 256 MB of heap bitmap when it only needs to reserve 128 MB. We halved the footprint without updating the calculation of bitmapSize on that next-to-last line. It's too late for Go 1.6 but we could do one of two things in Go 1.7:

(1) Halve the bitmapSize, since that's all we need for _MaxArena32 = 2GB.

(2) Double _MaxArena32. That's what the "Alternatively ..." comment above is suggesting, but there the assumption is it will require 512 MB of bitmap. Now that bitmaps are half as big, we can keep using the same 256 MB we're already reserving and use it to describe the entire 4 GB space, letting us eliminate this class of crash entirely.

I suspect we should do (2). But again it will need to wait for Go 1.7.

@rsc rsc modified the milestones: Go1.7, Go1.6Maybe Jan 27, 2016
@rsc
Copy link
Contributor

rsc commented Jan 27, 2016

/cc @RLH @aclements

@gopherbot
Copy link

CL https://golang.org/cl/18975 mentions this issue.

gopherbot pushed a commit that referenced this issue Jan 27, 2016
It's possible for arena_start+MaxArena32 to wrap.
We do the right thing in the bounds check but not in the print.

For #13992 (to fix the print there, not the bug).

Change-Id: I4df845d0c03f0f35461b128e4f6765d3ccb71c6d
Reviewed-on: https://go-review.googlesource.com/18975
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
@karalabe
Copy link
Contributor Author

karalabe commented Mar 8, 2016

We've found a bug in our code that used more memory than we anticipated it would, that may have caused running out of memory, and hitting some weird limits with the address space on 32 bit systems. If you wish you can close down this thread. I'll open a new one or post into this if the issue resurfaces.

@davecheney
Copy link
Contributor

Thanks. Please reopen if the situation changes.

@eliasnaur
Copy link
Contributor

@davecheney, are you sure? I think Russ' suggestions sounds good to me, eliminating this kind of crash entirely.

@davecheney
Copy link
Contributor

It's the OP's choice, see above.

@karalabe
Copy link
Contributor Author

karalabe commented Mar 8, 2016

I only said that we found a bug that increased the memory usage :P If there's a fix that would make the code more stable on 32 bit platforms I'm all for it :)

@gopherbot
Copy link

CL https://golang.org/cl/20370 mentions this issue.

@davecheney
Copy link
Contributor

If you wish you can close down this thread. I'll open a new one or post into this if the issue resurfaces.

Ok, my mistake. Reopening.

@davecheney davecheney reopened this Mar 8, 2016
@gopherbot
Copy link

CL https://golang.org/cl/20471 mentions this issue.

@ncopa
Copy link
Contributor

ncopa commented May 19, 2016

I think the fix should be pushed to 1.6 branch. Building docker with go 1.6.2 fails on arm but works with 1.6.1.

@aclements
Copy link
Member

@ncopa, the fix for this issue is quite invasive and it's fixing something that has been a problem for many Go releases (that is, the problem wasn't introduced in Go 1.6; it certainly wasn't introduced in Go 1.6.2), so I'm afraid it isn't really a candidate for cherry-picking to 1.6.

I'm surprised Docker built on ARM with 1.6.1 but not with 1.6.2. You could open a new issue about that with more details.

@ncopa
Copy link
Contributor

ncopa commented May 19, 2016

@aclements yes. I should have opened a new issue. I tried to backport the fix but it didnt solve the it. Must be something else that is happening.

@golang golang locked and limited conversation to collaborators May 19, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

8 participants