cmd/compile: out of memory with large slice and map composite literals #22010

h12w · 2017-09-25T05:19:05Z

What version of Go are you using (`go version`)?

go version go1.9 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/tmp.B05GgQUqzW/go"
GORACE=""
GOROOT="/usr/local/go1.9"
GOTOOLDIR="/usr/local/go1.9/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build029171386=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

go get h12.me/ua
go build -v h12.me/ua

What did you expect to see?

go build finishes as go1.8.3.

What did you see instead?

h12.me/ua
# h12.me/ua
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xb62665, 0x16)
	/usr/local/go/src/runtime/panic.go:605 +0x95
runtime.sysMap(0xc45a290000, 0x2f0000, 0xc420060100, 0xf05378)
	/usr/local/go/src/runtime/mem_linux.go:216 +0x1d0
runtime.(*mheap).sysAlloc(0xee3f40, 0x2f0000, 0xc42006fe20)
	/usr/local/go/src/runtime/malloc.go:470 +0xd7
runtime.(*mheap).grow(0xee3f40, 0x172, 0x0)
	/usr/local/go/src/runtime/mheap.go:887 +0x60
runtime.(*mheap).allocSpanLocked(0xee3f40, 0x172, 0xf05388, 0x7fe3c6a9fa18)
	/usr/local/go/src/runtime/mheap.go:800 +0x334
runtime.(*mheap).alloc_m(0xee3f40, 0x172, 0xffffffffffff0100, 0xc420060180)
	/usr/local/go/src/runtime/mheap.go:666 +0x118
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:733 +0x4d
runtime.systemstack(0xc42006ff08)
	/usr/local/go/src/runtime/asm_amd64.s:360 +0xab
runtime.(*mheap).alloc(0xee3f40, 0x172, 0x7fe3c6010100, 0x7fe3c6a9fa18)
	/usr/local/go/src/runtime/mheap.go:732 +0xa1
runtime.largeAlloc(0x2e2ba0, 0x7fe3c8220001, 0x7fe3c6a9fa18)
	/usr/local/go/src/runtime/malloc.go:827 +0x98
runtime.mallocgc.func1()
	/usr/local/go/src/runtime/malloc.go:722 +0x46
runtime.systemstack(0xc420020600)
	/usr/local/go/src/runtime/asm_amd64.s:344 +0x79
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1125

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:298 fp=0xc43ea636e8 sp=0xc43ea636e0 pc=0x455170
runtime.mallocgc(0x2e2ba0, 0xb2bf20, 0x17101, 0xc43c652228)
	/usr/local/go/src/runtime/malloc.go:721 +0x7b8 fp=0xc43ea63790 sp=0xc43ea636e8 pc=0x410638
runtime.makeslice(0xb2bf20, 0x1715d, 0x1715d, 0x1715d, 0x0, 0x80)
	/usr/local/go/src/runtime/slice.go:54 +0x77 fp=0xc43ea637c0 sp=0xc43ea63790 pc=0x43f157
cmd/compile/internal/ssa.makeLCArange(0xc43c652140, 0x0)
	/usr/local/go/src/cmd/compile/internal/ssa/lca.go:36 +0x8d fp=0xc43ea63920 sp=0xc43ea637c0 pc=0x5e783d
cmd/compile/internal/ssa.tighten(0xc43c652140)
	/usr/local/go/src/cmd/compile/internal/ssa/tighten.go:52 +0x2a6 fp=0xc43ea63a98 sp=0xc43ea63920 pc=0x8ff876
cmd/compile/internal/ssa.Compile(0xc43c652140)
	/usr/local/go/src/cmd/compile/internal/ssa/compile.go:70 +0x2bb fp=0xc43ea673d0 sp=0xc43ea63a98 pc=0x5cb19b
cmd/compile/internal/gc.buildssa(0xc43cae8b00, 0x0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:216 +0xd89 fp=0xc43ea67530 sp=0xc43ea673d0 pc=0x9f6b29
cmd/compile/internal/gc.compileSSA(0xc43cae8b00, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:240 +0x3c fp=0xc43ea675a8 sp=0xc43ea67530 pc=0x9c363c
cmd/compile/internal/gc.compile(0xc43cae8b00)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:219 +0x218 fp=0xc43ea67608 sp=0xc43ea675a8 pc=0x9c3568
cmd/compile/internal/gc.funccompile(0xc43cae8b00)
	/usr/local/go/src/cmd/compile/internal/gc/dcl.go:1049 +0xb7 fp=0xc43ea67660 sp=0xc43ea67608 pc=0x96a007
cmd/compile/internal/gc.fninit(0xc4203abe00, 0x28, 0x40)
	/usr/local/go/src/cmd/compile/internal/gc/init.go:207 +0xb59 fp=0xc43ea67880 sp=0xc43ea67660 pc=0x98fe39
cmd/compile/internal/gc.Main(0xb73f90)
	/usr/local/go/src/cmd/compile/internal/gc/main.go:592 +0x2c78 fp=0xc43ea67f08 sp=0xc43ea67880 pc=0x99e578
main.main()
	/usr/local/go/src/cmd/compile/main.go:49 +0x95 fp=0xc43ea67f80 sp=0xc43ea67f08 pc=0xabdfb5
runtime.main()
	/usr/local/go/src/runtime/proc.go:185 +0x20d fp=0xc43ea67fe0 sp=0xc43ea67f80 pc=0x42bedd
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc43ea67fe8 sp=0xc43ea67fe0 pc=0x457cf1

The text was updated successfully, but these errors were encountered:

mvdan · 2017-09-25T07:47:06Z

Both 1.9 and tip indeed seem to run out of memory. I have 7GB of free memory and it takes the build around 10s to use it all up, so it looks like it's stuck in an endless loop somewhere.

Will try to investigate and get a smaller reproduction program.

mvdan · 2017-09-25T08:01:53Z

Nope - removing the 23k elements from the handsets map fixes it on my machine. Removing all elements of deviceInfoMap also fixes it. But leaving both eats up my memory.

I think I saw issues about huge composite literals (especially really long slices and maps) using too much CPU time or memory. I wonder what could have caused the regression from 1.8. A bit beyond me, I'm afraid.

CC @mdempsky @randall77

gopherbot · 2017-09-25T17:34:24Z

Change https://golang.org/cl/66050 mentions this issue: cmd/compile: improve static map initialization

randall77 · 2017-09-25T17:37:27Z

CL 66050 reduces memory use from ~15GB to ~0.5GB.

ianlancetaylor · 2017-09-25T18:04:46Z

That is only 96% smaller. Why stop there?

mdempsky · 2017-09-25T18:43:47Z

@ianlancetaylor Gotta leave some work for 1.11.

randall77 · 2017-09-25T19:01:05Z

This is now fixed on tip.
What I don't understand is why 1.8 is better than 1.9.
1.8 uses mapassign and 1.9 uses mapassign_faststr, but I don't see how that would make much of a difference. The generated assembly is otherwise very similar.

Since this is a regression from 1.8 to 1.9, there's a possibility we could backport this fix (or another fix more directly related to the 1.8->1.9 regression, if we can understand it) to 1.9.1. But I'm leaning against doing this, it is a difficult bug to trigger (probably requires generated code) and can be worked around (doing the optimization in CL66050 manually).

mvdan added this to the Go1.10 milestone Sep 25, 2017

mvdan changed the title ~~go build out of memory~~ cmd/compile: out of memory with large slice and map composite literals Sep 25, 2017

gopherbot closed this as completed in 91cb9ed Sep 25, 2017

randall77 mentioned this issue Oct 16, 2017

cmd/compile: Go 1.9 oom killed on compilation #21713

Closed

golang locked and limited conversation to collaborators Sep 25, 2018

gopherbot added the FrozenDueToAge label Sep 25, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/compile: out of memory with large slice and map composite literals #22010

cmd/compile: out of memory with large slice and map composite literals #22010

h12w commented Sep 25, 2017

mvdan commented Sep 25, 2017

mvdan commented Sep 25, 2017

gopherbot commented Sep 25, 2017

randall77 commented Sep 25, 2017

ianlancetaylor commented Sep 25, 2017

mdempsky commented Sep 25, 2017

randall77 commented Sep 25, 2017

cmd/compile: out of memory with large slice and map composite literals #22010

cmd/compile: out of memory with large slice and map composite literals #22010

Comments

h12w commented Sep 25, 2017

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

mvdan commented Sep 25, 2017

mvdan commented Sep 25, 2017

gopherbot commented Sep 25, 2017

randall77 commented Sep 25, 2017

ianlancetaylor commented Sep 25, 2017

mdempsky commented Sep 25, 2017

randall77 commented Sep 25, 2017

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?