Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: arena mapping creates large core files and bad mlockall behavior #28081

Open
randall77 opened this issue Oct 8, 2018 · 16 comments
Open

runtime: arena mapping creates large core files and bad mlockall behavior #28081

randall77 opened this issue Oct 8, 2018 · 16 comments
Labels
NeedsFix The path to resolution is known, but the work has not been done. Performance
Milestone
Unplanned

Comments

@randall77
Copy link
Contributor 

package main

var p *int

func main() {
	*p = 0
}

Run with

$ ulimit -c unlimited
$ GOTRACEBACK=crash go run test.go
$ ls -l core

Go 1.11 and tip generate a ~100MB core. Go 1.10 and earlier only generate a ~2MB core.
Probably related to the arena changes in 1.11.

@aclements @heschik @hyangah
@mknyszek
Copy link
Contributor

mknyszek commented Oct 8, 2018

Reproduced locally, but if I stat the file I'm pretty sure it's appearing as sparse.

While the reported size is 100MB, (# of blocks) * (# of bytes per block) = 8640 * 4096 ~= 34K for me.

@aclements
Copy link
Member

Almost certainly related to the arena changes.

These core files are large, but are very sparse, so their actual on-disk size is much smaller.

@aclements
Copy link
Member

If we care about fixing this, we probably need to map the arenas PROT_NONE initially and incrementally re-map them read/write. That means we can't just make a giant span out of each new arena, but perhaps that's fine. We'd have to keep track of how far we'd mapped, but if we're willing to only keep the latest arena partially mapped, that could be a single value in mheap.

@randall77
Copy link
Contributor Author

Ok, it sounds like disk space is not a huge issue.
The core dump reader does mmap the file, so it burns address space in the reader. That's not a big deal, as long as it's not also burning physical memory. Is mmap smart enough to share zero pages when mmap'ing a sparse file? I hope so.
(The reader should ideally never access the zero pages, so linux might never get to that decision point.)

@aclements
Copy link
Member

Huh, it's not smart enough to zero-share those pages, even if it's a read-only shared mapping.

package main

import (
	"log"
	"os"
	"syscall"
	"time"
)

func main() {
	const size = 512 << 20
	f, _ := os.Create("/tmp/sparse")
	f.Truncate(size)
	p, _ := syscall.Mmap(int(f.Fd()), 0, size, syscall.PROT_READ, syscall.MAP_SHARED)
	sum := 0
	for i := range p {
		sum += int(p[i])
	}
	if sum != 0 {
		log.Fatal(sum)
	}
	time.Sleep(10 * time.Second)
}

@randall77 randall77 mentioned this issue Oct 10, 2018
Closed
@gaolizhou
Copy link

gaolizhou commented Oct 11, 2018 •
edited

Hi Guys, Here is the backtrace for the 100M memory malloc.

[root@10-25-51-148 ~]# pmap -x `pgrep mem-test`
17590:   ./mem-test
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     316     244       0 r-x-- mem-test
000000000044f000     416     100       0 r---- mem-test
00000000004b7000      12      12      12 rw--- mem-test
00000000004ba000     124      36      36 rw---   [ anon ]
000000c000000000   65536    2048    2048 rw---   [ anon ]
00007f0f11962000   35264    2168    2168 rw---   [ anon ]
00007ffe11d76000     132      16      16 rw---   [ stack ]
00007ffe11dec000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- ------- 
total kB          101812    4628    4280

(gdb) bt
#0  runtime.sysMmap () at /home/lizgao/go/src/runtime/sys_linux_amd64.s:450
#1  0x0000000000403261 in runtime.mmap (addr=0xc000000000, fd=-1, flags=34, n=67108864, off=0, prot=0, ~r6=, ~r7=)
    at /home/lizgao/go/src/runtime/cgo_mmap.go:39
#2  0x0000000000410d6e in runtime.sysReserve (n=, v=, ~r2=) at /home/lizgao/go/src/runtime/mem_linux.go:144
#3  0x00000000004091b9 in runtime.(*mheap).sysAlloc (h=0x4be660 , n=67108864, size=0, v=0x0) at /home/lizgao/go/src/runtime/malloc.go:541
#4  0x000000000041c372 in runtime.(*mheap).grow (h=0x4be660 , npage=, ~r1=) at /home/lizgao/go/src/runtime/mheap.go:919
#5  0x000000000041c1f7 in runtime.(*mheap).allocSpanLocked (h=0x4be660 , npage=1, stat=0x4d6c68 , ~r2=)
    at /home/lizgao/go/src/runtime/mheap.go:848
#6  0x000000000041ba09 in runtime.(*mheap).alloc_m (h=0x4be660 , large=false, npage=1, spanclass=42 '*', ~r3=)
    at /home/lizgao/go/src/runtime/mheap.go:692
#7  0x00000000004451ac in runtime.(*mheap).alloc.func1 () at /home/lizgao/go/src/runtime/mheap.go:759
#8  0x000000000041bcaa in runtime.(*mheap).alloc (h=, large=, needzero=, npage=, 
    spanclass=, ~r4=) at /home/lizgao/go/src/runtime/mheap.go:758
#9  0x0000000000410874 in runtime.(*mcentral).grow (c=0x4c0418 , ~r0=) at /home/lizgao/go/src/runtime/mcentral.go:232
#10 0x00000000004103c8 in runtime.(*mcentral).cacheSpan (c=0x4c0418 , ~r0=) at /home/lizgao/go/src/runtime/mcentral.go:106
#11 0x000000000040ff85 in runtime.(*mcache).refill (c=0x7ffff7fbd000, spc=42 '*') at /home/lizgao/go/src/runtime/mcache.go:118
#12 0x0000000000444692 in runtime.(*mcache).nextFree.func1 () at /home/lizgao/go/src/runtime/malloc.go:749
#13 0x0000000000409536 in runtime.(*mcache).nextFree (c=0x7ffff7fbd000, s=, shouldhelpgc=, spc=42 '*', v=)
    at /home/lizgao/go/src/runtime/malloc.go:748
#14 0x0000000000409e83 in runtime.mallocgc (needzero=true, size=384, typ=0x46baa0, ~r3=) at /home/lizgao/go/src/runtime/malloc.go:903
#15 0x000000000040a268 in runtime.newobject (typ=, ~r1=) at /home/lizgao/go/src/runtime/malloc.go:1032
#16 0x000000000042b411 in runtime.malg (stacksize=, ~r1=) at /home/lizgao/go/src/runtime/proc.go:3285
#17 0x0000000000420649 in runtime.mpreinit (mp=) at /home/lizgao/go/src/runtime/os_linux.go:311
#18 0x0000000000424ff1 in runtime.mcommoninit (mp=0x4ba3a0 ) at /home/lizgao/go/src/runtime/proc.go:624
#19 0x0000000000424cb9 in runtime.schedinit () at /home/lizgao/go/src/runtime/proc.go:546
#20 0x0000000000446b8a in runtime.rt0_go () at /home/lizgao/go/src/runtime/asm_amd64.s:195
#21 0x0000000000000001 in ?? ()
#22 0x00007fffffffe578 in ?? ()
#23 0x0000000000000001 in ?? ()
#24 0x00007fffffffe578 in ?? ()
#25 0x0000000000000000 in ?? ()
(gdb) c
Continuing.

Breakpoint 1, runtime.sysMmap () at /home/lizgao/go/src/runtime/sys_linux_amd64.s:450
450             MOVQ    addr+0(FP), DI
(gdb) bt
#0  runtime.sysMmap () at /home/lizgao/go/src/runtime/sys_linux_amd64.s:450
#1  0x0000000000403261 in runtime.mmap (addr=0xc000000000, fd=-1, flags=50, n=67108864, off=0, prot=3, ~r6=, ~r7=)
    at /home/lizgao/go/src/runtime/cgo_mmap.go:39
#2  0x0000000000410e1a in runtime.sysMap (n=, sysStat=, v=0xc000000000) at /home/lizgao/go/src/runtime/mem_linux.go:154
#3  0x0000000000408e87 in runtime.(*mheap).sysAlloc (h=0x4be660 , n=67108864, size=, v=)
    at /home/lizgao/go/src/runtime/malloc.go:619
#4  0x000000000041c372 in runtime.(*mheap).grow (h=0x4be660 , npage=, ~r1=) at /home/lizgao/go/src/runtime/mheap.go:919
#5  0x000000000041c1f7 in runtime.(*mheap).allocSpanLocked (h=0x4be660 , npage=1, stat=0x4d6c68 , ~r2=)
    at /home/lizgao/go/src/runtime/mheap.go:848
#6  0x000000000041ba09 in runtime.(*mheap).alloc_m (h=0x4be660 , large=false, npage=1, spanclass=42 '*', ~r3=)
    at /home/lizgao/go/src/runtime/mheap.go:692
#7  0x00000000004451ac in runtime.(*mheap).alloc.func1 () at /home/lizgao/go/src/runtime/mheap.go:759
#8  0x000000000041bcaa in runtime.(*mheap).alloc (h=, large=, needzero=, npage=, 
    spanclass=, ~r4=) at /home/lizgao/go/src/runtime/mheap.go:758
#9  0x0000000000410874 in runtime.(*mcentral).grow (c=0x4c0418 , ~r0=) at /home/lizgao/go/src/runtime/mcentral.go:232
#10 0x00000000004103c8 in runtime.(*mcentral).cacheSpan (c=0x4c0418 , ~r0=) at /home/lizgao/go/src/runtime/mcentral.go:106
#11 0x000000000040ff85 in runtime.(*mcache).refill (c=0x7ffff7fbd000, spc=42 '*') at /home/lizgao/go/src/runtime/mcache.go:118
#12 0x0000000000444692 in runtime.(*mcache).nextFree.func1 () at /home/lizgao/go/src/runtime/malloc.go:749
#13 0x0000000000409536 in runtime.(*mcache).nextFree (c=0x7ffff7fbd000, s=, shouldhelpgc=, spc=42 '*', v=)
    at /home/lizgao/go/src/runtime/malloc.go:748
#14 0x0000000000409e83 in runtime.mallocgc (needzero=true, size=384, typ=0x46baa0, ~r3=) at /home/lizgao/go/src/runtime/malloc.go:903
#15 0x000000000040a268 in runtime.newobject (typ=, ~r1=) at /home/lizgao/go/src/runtime/malloc.go:1032
#16 0x000000000042b411 in runtime.malg (stacksize=, ~r1=) at /home/lizgao/go/src/runtime/proc.go:3285
#17 0x0000000000420649 in runtime.mpreinit (mp=) at /home/lizgao/go/src/runtime/os_linux.go:311
#18 0x0000000000424ff1 in runtime.mcommoninit (mp=0x4ba3a0 ) at /home/lizgao/go/src/runtime/proc.go:624
#19 0x0000000000424cb9 in runtime.schedinit () at /home/lizgao/go/src/runtime/proc.go:546
#20 0x0000000000446b8a in runtime.rt0_go () at /home/lizgao/go/src/runtime/asm_amd64.s:195
#21 0x0000000000000001 in ?? ()
#22 0x00007fffffffe578 in ?? ()
#23 0x0000000000000001 in ?? ()
#24 0x00007fffffffe578 in ?? ()
#25 0x0000000000000000 in ?? ()
(gdb) c
Continuing.

Breakpoint 1, runtime.sysMmap () at /home/lizgao/go/src/runtime/sys_linux_amd64.s:450
450             MOVQ    addr+0(FP), DI
(gdb) bt
#0  runtime.sysMmap () at /home/lizgao/go/src/runtime/sys_linux_amd64.s:450
#1  0x0000000000403261 in runtime.mmap (addr=0x0, fd=-1, flags=34, n=33554432, off=0, prot=3, ~r6=, ~r7=)
    at /home/lizgao/go/src/runtime/cgo_mmap.go:39
#2  0x000000000041099e in runtime.sysAlloc (n=33554432, sysStat=, ~r2=) at /home/lizgao/go/src/runtime/mem_linux.go:21
#3  0x000000000040a899 in runtime.persistentalloc1 (align=67108864, size=33554432, sysStat=0x0, ~r3=) at /home/lizgao/go/src/runtime/malloc.go:1172
#4  0x0000000000444765 in runtime.persistentalloc.func1 () at /home/lizgao/go/src/runtime/malloc.go:1143
#5  0x000000000040a632 in runtime.persistentalloc (align=, size=, sysStat=, ~r3=)
    at /home/lizgao/go/src/runtime/malloc.go:1142
#6  0x0000000000408fce in runtime.(*mheap).sysAlloc (h=0x4be660 , n=67108864, size=, v=)
    at /home/lizgao/go/src/runtime/malloc.go:627
#7  0x000000000041c372 in runtime.(*mheap).grow (h=0x4be660 , npage=, ~r1=) at /home/lizgao/go/src/runtime/mheap.go:919
#8  0x000000000041c1f7 in runtime.(*mheap).allocSpanLocked (h=0x4be660 , npage=1, stat=0x4d6c68 , ~r2=)
    at /home/lizgao/go/src/runtime/mheap.go:848
#9  0x000000000041ba09 in runtime.(*mheap).alloc_m (h=0x4be660 , large=false, npage=1, spanclass=42 '*', ~r3=)
    at /home/lizgao/go/src/runtime/mheap.go:692
#10 0x00000000004451ac in runtime.(*mheap).alloc.func1 () at /home/lizgao/go/src/runtime/mheap.go:759
#11 0x000000000041bcaa in runtime.(*mheap).alloc (h=, large=, needzero=, npage=, 
    spanclass=, ~r4=) at /home/lizgao/go/src/runtime/mheap.go:758
#12 0x0000000000410874 in runtime.(*mcentral).grow (c=0x4c0418 , ~r0=) at /home/lizgao/go/src/runtime/mcentral.go:232
#13 0x00000000004103c8 in runtime.(*mcentral).cacheSpan (c=0x4c0418 , ~r0=) at /home/lizgao/go/src/runtime/mcentral.go:106
#14 0x000000000040ff85 in runtime.(*mcache).refill (c=0x7ffff7fbd000, spc=42 '*') at /home/lizgao/go/src/runtime/mcache.go:118
#15 0x0000000000444692 in runtime.(*mcache).nextFree.func1 () at /home/lizgao/go/src/runtime/malloc.go:749
#16 0x0000000000409536 in runtime.(*mcache).nextFree (c=0x7ffff7fbd000, s=, shouldhelpgc=, spc=42 '*', v=)
    at /home/lizgao/go/src/runtime/malloc.go:748
#17 0x0000000000409e83 in runtime.mallocgc (needzero=true, size=384, typ=0x46baa0, ~r3=) at /home/lizgao/go/src/runtime/malloc.go:903
#18 0x000000000040a268 in runtime.newobject (typ=, ~r1=) at /home/lizgao/go/src/runtime/malloc.go:1032
#19 0x000000000042b411 in runtime.malg (stacksize=, ~r1=) at /home/lizgao/go/src/runtime/proc.go:3285
#20 0x0000000000420649 in runtime.mpreinit (mp=) at /home/lizgao/go/src/runtime/os_linux.go:311
#21 0x0000000000424ff1 in runtime.mcommoninit (mp=0x4ba3a0 ) at /home/lizgao/go/src/runtime/proc.go:624
#22 0x0000000000424cb9 in runtime.schedinit () at /home/lizgao/go/src/runtime/proc.go:546
#23 0x0000000000446b8a in runtime.rt0_go () at /home/lizgao/go/src/runtime/asm_amd64.s:195
#24 0x0000000000000001 in ?? ()
#25 0x00007fffffffe578 in ?? ()
#26 0x0000000000000001 in ?? ()
#27 0x00007fffffffe578 in ?? ()
#28 0x0000000000000000 in ?? ()
(gdb) c
Continuing.

Breakpoint 1, runtime.sysMmap () at /home/lizgao/go/src/runtime/sys_linux_amd64.s:450
450             MOVQ    addr+0(FP), DI
(gdb) bt
#0  runtime.sysMmap () at /home/lizgao/go/src/runtime/sys_linux_amd64.s:450
#1  0x0000000000403261 in runtime.mmap (addr=0x0, fd=-1, flags=34, n=2162688, off=0, prot=3, ~r6=, ~r7=)
    at /home/lizgao/go/src/runtime/cgo_mmap.go:39
#2  0x000000000041099e in runtime.sysAlloc (n=2162688, sysStat=, ~r2=) at /home/lizgao/go/src/runtime/mem_linux.go:21
#3  0x000000000040a899 in runtime.persistentalloc1 (align=67108864, size=2162688, sysStat=0x4d6cb8 , ~r3=)
    at /home/lizgao/go/src/runtime/malloc.go:1172
#4  0x0000000000444765 in runtime.persistentalloc.func1 () at /home/lizgao/go/src/runtime/malloc.go:1143
#5  0x000000000040a632 in runtime.persistentalloc (align=, size=, sysStat=, ~r3=)
    at /home/lizgao/go/src/runtime/malloc.go:1142
#6  0x0000000000408f9c in runtime.(*mheap).sysAlloc (h=0x4be660 , n=67108864, size=, v=)
    at /home/lizgao/go/src/runtime/malloc.go:640
#7  0x000000000041c372 in runtime.(*mheap).grow (h=0x4be660 , npage=, ~r1=) at /home/lizgao/go/src/runtime/mheap.go:919
#8  0x000000000041c1f7 in runtime.(*mheap).allocSpanLocked (h=0x4be660 , npage=1, stat=0x4d6c68 , ~r2=)
    at /home/lizgao/go/src/runtime/mheap.go:848
#9  0x000000000041ba09 in runtime.(*mheap).alloc_m (h=0x4be660 , large=false, npage=1, spanclass=42 '*', ~r3=)
    at /home/lizgao/go/src/runtime/mheap.go:692
#10 0x00000000004451ac in runtime.(*mheap).alloc.func1 () at /home/lizgao/go/src/runtime/mheap.go:759
#11 0x000000000041bcaa in runtime.(*mheap).alloc (h=, large=, needzero=, npage=, 
    spanclass=, ~r4=) at /home/lizgao/go/src/runtime/mheap.go:758
#12 0x0000000000410874 in runtime.(*mcentral).grow (c=0x4c0418 , ~r0=) at /home/lizgao/go/src/runtime/mcentral.go:232
#13 0x00000000004103c8 in runtime.(*mcentral).cacheSpan (c=0x4c0418 , ~r0=) at /home/lizgao/go/src/runtime/mcentral.go:106
#14 0x000000000040ff85 in runtime.(*mcache).refill (c=0x7ffff7fbd000, spc=42 '*') at /home/lizgao/go/src/runtime/mcache.go:118
#15 0x0000000000444692 in runtime.(*mcache).nextFree.func1 () at /home/lizgao/go/src/runtime/malloc.go:749
#16 0x0000000000409536 in runtime.(*mcache).nextFree (c=0x7ffff7fbd000, s=, shouldhelpgc=, spc=42 '*', v=)
    at /home/lizgao/go/src/runtime/malloc.go:748
#17 0x0000000000409e83 in runtime.mallocgc (needzero=true, size=384, typ=0x46baa0, ~r3=) at /home/lizgao/go/src/runtime/malloc.go:903
#18 0x000000000040a268 in runtime.newobject (typ=, ~r1=) at /home/lizgao/go/src/runtime/malloc.go:1032
#19 0x000000000042b411 in runtime.malg (stacksize=, ~r1=) at /home/lizgao/go/src/runtime/proc.go:3285
#20 0x0000000000420649 in runtime.mpreinit (mp=) at /home/lizgao/go/src/runtime/os_linux.go:311
#21 0x0000000000424ff1 in runtime.mcommoninit (mp=0x4ba3a0 ) at /home/lizgao/go/src/runtime/proc.go:624
#22 0x0000000000424cb9 in runtime.schedinit () at /home/lizgao/go/src/runtime/proc.go:546
#23 0x0000000000446b8a in runtime.rt0_go () at /home/lizgao/go/src/runtime/asm_amd64.s:195
#24 0x0000000000000001 in ?? ()
#25 0x00007fffffffe578 in ?? ()
#26 0x0000000000000001 in ?? ()
#27 0x00007fffffffe578 in ?? ()
#28 0x0000000000000000 in ?? ()
(gdb)

@aclements
Copy link
Member

@gaolizhou, thanks for the backtrace. We do understand exactly what's causing this issue. It's just an unintended consequence of a very intentional change that switched from mapping the heap in small increments to mapping it in 64MB increments.

@bcmills
Copy link
Contributor

bcmills commented Oct 23, 2018

We do understand exactly what's causing this issue. It's just an unintended consequence of a very intentional change that switched from mapping the heap in small increments to mapping it in 64MB increments.

Do we have a fix in mind? If not, does the benefit of that change outweigh the regression in core sizes?

@bcmills bcmills added this to the Go1.12 milestone Oct 23, 2018
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 23, 2018
@aclements
Copy link
Member

Do we have a fix in mind? If not, does the benefit of that change outweigh the regression in core sizes?

Map arenas PROT_NONE initially and incrementally re-map them PROT_READ|PROT_WRITE as we need more space. We only need to keep track of the "current arena" and how far we've mapped it (any other arena can be fully mapped).

I don't think this is very hard to implement or adds much complexity, though everything in sysAlloc is a little tricky.

@bcmills bcmills added the NeedsFix The path to resolution is known, but the work has not been done. label Oct 23, 2018
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 23, 2018
@aclements aclements modified the milestones: Go1.12, Go1.13 Jan 8, 2019
@aclements aclements changed the title runtime: large core files runtime: arena mapping creates large core files and bad mlockall behavior Jan 8, 2019
@gopherbot
Copy link

Change https://golang.org/cl/175497 mentions this issue: internal/gocore: update to handle 1.12 core files

gopherbot pushed a commit to golang/debug that referenced this issue May 15, 2019
@randall77
internal/gocore: update to handle 1.12 core files
621e2d3 
Just a few adjustments to handle runtime changes.

Use zip files for test data so we don't use so much
disk space for large core files. See golang/go#28081.

Fixes golang/go#30631

Change-Id: I6a11aae56e715540fc320cbe70e426eeafa2f64d
Reviewed-on: https://go-review.googlesource.com/c/debug/+/175497
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
@aclements aclements modified the milestones: Go1.13, Unplanned Jun 25, 2019
@johnsom
Copy link

johnsom commented Aug 16, 2019 •
edited

We do understand exactly what's causing this issue. It's just an unintended consequence of a very intentional change that switched from mapping the heap in small increments to mapping it in 64MB increments.

Is there a way to switch this back to the old behavior either at compile or runtime? We have a light weight application that we had been running in a 64MB sized VM with no issue. With this change, when we build it on the newer tool chain (>= 1.11) it won't start in those small VMs. We get "fatal error: runtime: out of memory".

@aclements
Copy link
Member

@johnsom, no, there's is no way to switch this behavior. But passing MCL_ONFAULT when you call mlockall should fix it. This is a good idea anyway: the default behavior of mlockall turns an otherwise free resource (virtual address space) into an expensive resource (physical memory). And since this is a process-global operation, it globally interferes with engineering trade-offs based on these relative resource costs.

@johnsom
Copy link

johnsom commented Aug 16, 2019

Sadly our code does not call mlockall at all, it only uses libraries.
The code is here: https://github.com/openstack/octavia-tempest-plugin/blob/master/octavia_tempest_plugin/contrib/httpd/httpd.go

The trace is here:

`fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x6b8264, 0x16)
/usr/lib/golang/src/runtime/panic.go:608 +0x72 fp=0x7fffd923fae0 sp=0x7fffd923fab0 pc=0x429f62
runtime.sysMap(0xc000000000, 0x4000000, 0x8cc258)
/usr/lib/golang/src/runtime/mem_linux.go:156 +0xc7 fp=0x7fffd923fb20 sp=0x7fffd923fae0 pc=0x416567
runtime.(*mheap).sysAlloc(0x8b3080, 0x4000000, 0x0, 0x0)
/usr/lib/golang/src/runtime/malloc.go:619 +0x1c7 fp=0x7fffd923fba8 sp=0x7fffd923fb20 pc=0x40a837
runtime.(*mheap).grow(0x8b3080, 0x1, 0x0)
/usr/lib/golang/src/runtime/mheap.go:920 +0x42 fp=0x7fffd923fc00 sp=0x7fffd923fba8 pc=0x422762
runtime.(*mheap).allocSpanLocked(0x8b3080, 0x1, 0x8cc268, 0x0)
/usr/lib/golang/src/runtime/mheap.go:848 +0x337 fp=0x7fffd923fc40 sp=0x7fffd923fc00 pc=0x4225e7
runtime.(*mheap).alloc_m(0x8b3080, 0x1, 0x2a, 0x0)
/usr/lib/golang/src/runtime/mheap.go:692 +0x119 fp=0x7fffd923fc80 sp=0x7fffd923fc40 pc=0x421df9
runtime.(*mheap).alloc.func1()
/usr/lib/golang/src/runtime/mheap.go:759 +0x4c fp=0x7fffd923fcb8 sp=0x7fffd923fc80 pc=0x4524bc
runtime.(*mheap).alloc(0x8b3080, 0x1, 0x1002a, 0x0)
/usr/lib/golang/src/runtime/mheap.go:758 +0x8a fp=0x7fffd923fd08 sp=0x7fffd923fcb8 pc=0x42209a
runtime.(*mcentral).grow(0x8b4e38, 0x0)
/usr/lib/golang/src/runtime/mcentral.go:232 +0x94 fp=0x7fffd923fd50 sp=0x7fffd923fd08 pc=0x415f64
runtime.(*mcentral).cacheSpan(0x8b4e38, 0x7f824abe7000)
/usr/lib/golang/src/runtime/mcentral.go:106 +0x2f8 fp=0x7fffd923fd98 sp=0x7fffd923fd50 pc=0x415ab8
runtime.(*mcache).refill(0x7f824abe7000, 0x2a)
/usr/lib/golang/src/runtime/mcache.go:122 +0x95 fp=0x7fffd923fdc8 sp=0x7fffd923fd98 pc=0x415675
runtime.(*mcache).nextFree.func1()
/usr/lib/golang/src/runtime/malloc.go:749 +0x32 fp=0x7fffd923fde8 sp=0x7fffd923fdc8 pc=0x4518d2
runtime.(*mcache).nextFree(0x7f824abe7000, 0x8cc22a, 0x4000, 0x7f824abe7000, 0x7fffd923fea8)
/usr/lib/golang/src/runtime/malloc.go:748 +0xb6 fp=0x7fffd923fe40 sp=0x7fffd923fde8 pc=0x40aee6
runtime.mallocgc(0x180, 0x6b19e0, 0x7fffd923ff01, 0x7f824abeb000)
/usr/lib/golang/src/runtime/malloc.go:903 +0x793 fp=0x7fffd923fee0 sp=0x7fffd923fe40 pc=0x40b833
runtime.newobject(0x6b19e0, 0x8cc2c0)
/usr/lib/golang/src/runtime/malloc.go:1032 +0x38 fp=0x7fffd923ff10 sp=0x7fffd923fee0 pc=0x40bc18
runtime.malg(0x7f8200008000, 0x7f824abe7000)
/usr/lib/golang/src/runtime/proc.go:3285 +0x31 fp=0x7fffd923ff50 sp=0x7fffd923ff10 pc=0x433301
runtime.mpreinit(0x8adb60)
/usr/lib/golang/src/runtime/os_linux.go:311 +0x29 fp=0x7fffd923ff70 sp=0x7fffd923ff50 pc=0x428399
runtime.mcommoninit(0x8adb60)
/usr/lib/golang/src/runtime/proc.go:624 +0xc1 fp=0x7fffd923ffa8 sp=0x7fffd923ff70 pc=0x42cdb1
runtime.schedinit()
/usr/lib/golang/src/runtime/proc.go:546 +0x89 fp=0x7fffd9240010 sp=0x7fffd923ffa8 pc=0x42ca79
runtime.rt0_go(0x7fffd9240048, 0x1, 0x7fffd9240048, 0x0, 0x0, 0x1, 0x7fffd9241e6a, 0x0, 0x7fffd9241e7b, 0x7fffd9241e98, ...)
/usr/lib/golang/src/runtime/asm_amd64.s:195 +0x11a fp=0x7fffd9240018 sp=0x7fffd9240010 pc=0x45412a`

@checkmatesolo

This comment has been minimized.

Sign in to view
@mknyszek
Copy link
Contributor

mknyszek commented Sep 6, 2019

Is there a way to switch this back to the old behavior either at compile or runtime? We have a light weight application that we had been running in a 64MB sized VM with no issue. With this change, when we build it on the newer tool chain (>= 1.11) it won't start in those small VMs. We get "fatal error: runtime: out of memory".

Is this a Linux VM? If so, simply mapping more memory shouldn't cause such problems, unless you have ulimit -v set low or overcommit configured conservatively. Are either of these the case for you?

It's possible you're seeing this for other reasons. For example, the RSS of your application increased with newer versions of Go, which could be a legitimate bug but distinct from anything in this issue.

@egodigitus
Copy link

But passing MCL_ONFAULT when you call mlockall should fix it.

Seems like the current go syscall is not supporting MCL_ONFAULT according to
https://golang.org/pkg/syscall/#pkg-constants

@aclements aclements mentioned this issue Nov 16, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsFix The path to resolution is known, but the work has not been done. Performance
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
9 participants
@mknyszek @aclements @gaolizhou @bcmills @randall77 @gopherbot @johnsom @egodigitus @checkmatesolo