cmd/compile: runs out of memory building github.com/yuin/goldmark/util #41058

zhangguanzhang · 2020-08-27T02:08:44Z

What version of Go are you using (`go version`)?

$ go version
go version go1.14.7 linux/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/GO"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,https://mirrors.aliyun.com/goproxy/,https://goproxy.io,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="0"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build355318494=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ git clone https://github.com/gohugoio/hugo.git
$ cd hugo
$ go build -o hugo -gcflags='all=-trimpath=/root/hugo -N -l' -asmflags 'all=-trimpath=/root/hugo' main.go
# github.com/yuin/goldmark/util
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xe84eda, 0x16)
	/usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.sysMap(0xc0c4000000, 0x4000000, 0x1667158)
	/usr/local/go/src/runtime/mem_linux.go:169 +0xc5
runtime.(*mheap).sysAlloc(0x1650bc0, 0x400000, 0x1650bc8, 0x16)
	/usr/local/go/src/runtime/malloc.go:715 +0x1cd
runtime.(*mheap).grow(0x1650bc0, 0x16, 0x0)
	/usr/local/go/src/runtime/mheap.go:1286 +0x11c
runtime.(*mheap).allocSpan(0x1650bc0, 0x16, 0x7f50163d0100, 0x1667168, 0x414db6)
	/usr/local/go/src/runtime/mheap.go:1124 +0x6a0
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:871 +0x64
runtime.(*mheap).alloc(0x1650bc0, 0x16, 0x7f50163d0101, 0x7f5018ce33e8)
	/usr/local/go/src/runtime/mheap.go:865 +0x81
runtime.largeAlloc(0x2b190, 0xc0003d0101, 0x7f5018ce33e8)
	/usr/local/go/src/runtime/malloc.go:1152 +0x92
runtime.mallocgc.func1()
	/usr/local/go/src/runtime/malloc.go:1047 +0x46
runtime.systemstack(0x0)
	/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1041

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc00f20f248 sp=0xc00f20f240 pc=0x4628e0
runtime.mallocgc(0x2b190, 0xe2efc0, 0xffffff0000010401, 0xc001f461c0)
	/usr/local/go/src/runtime/malloc.go:1046 +0x895 fp=0xc00f20f2e8 sp=0xc00f20f248 pc=0x40c725
runtime.makeslice(0xe2efc0, 0x0, 0x2b19, 0xc0c3fba000)
	/usr/local/go/src/runtime/slice.go:49 +0x6c fp=0xc00f20f318 sp=0xc00f20f2e8 pc=0x44a32c
cmd/compile/internal/ssa.(*regAllocState).computeLive(0xc008a12300)
	/usr/local/go/src/cmd/compile/internal/ssa/regalloc.go:2479 +0x17a3 fp=0xc00f20f678 sp=0xc00f20f318 pc=0x692513
cmd/compile/internal/ssa.(*regAllocState).init(0xc008a12300, 0xc0015c2000)
	/usr/local/go/src/cmd/compile/internal/ssa/regalloc.go:665 +0x681 fp=0xc00f20f7a8 sp=0xc00f20f678 pc=0x682e11
cmd/compile/internal/ssa.regalloc(0xc0015c2000)
	/usr/local/go/src/cmd/compile/internal/ssa/regalloc.go:145 +0x4a fp=0xc00f20f7d0 sp=0xc00f20f7a8 pc=0x6809ca
cmd/compile/internal/ssa.Compile(0xc0015c2000)
	/usr/local/go/src/cmd/compile/internal/ssa/compile.go:92 +0x9a5 fp=0xc00f213410 sp=0xc00f20f7d0 pc=0x6343e5
cmd/compile/internal/gc.buildssa(0xc00126fa20, 0x0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:445 +0xcd8 fp=0xc00f2135a0 sp=0xc00f213410 pc=0xce7a08
cmd/compile/internal/gc.compileSSA(0xc00126fa20, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:298 +0x5d fp=0xc00f213668 sp=0xc00f2135a0 pc=0xcb06dd
cmd/compile/internal/gc.compile(0xc00126fa20)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:277 +0x33b fp=0xc00f2136e0 sp=0xc00f213668 pc=0xcb061b
cmd/compile/internal/gc.funccompile(0xc00126fa20)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:222 +0xc1 fp=0xc00f213738 sp=0xc00f2136e0 pc=0xcb01d1
cmd/compile/internal/gc.fninit(0xc00045e480, 0x4b, 0x90)
	/usr/local/go/src/cmd/compile/internal/gc/init.go:63 +0x49e fp=0xc00f213858 sp=0xc00f213738 pc=0xc7592e
cmd/compile/internal/gc.Main(0xe9d7c0)
	/usr/local/go/src/cmd/compile/internal/gc/main.go:721 +0x3aed fp=0xc00f213f10 sp=0xc00f213858 pc=0xc88add
main.main()
	/usr/local/go/src/cmd/compile/main.go:52 +0xac fp=0xc00f213f88 sp=0xc00f213f10 pc=0xdc776c
runtime.main()
	/usr/local/go/src/runtime/proc.go:203 +0x1fa fp=0xc00f213fe0 sp=0xc00f213f88 pc=0x4357fa
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00f213fe8 sp=0xc00f213fe0 pc=0x464871

What did you expect to see?

What did you see instead?

The text was updated successfully, but these errors were encountered:

davecheney · 2020-08-27T02:18:54Z

How much memory does this system have available? Are you building inside a system like docker?

zhangguanzhang · 2020-08-27T03:03:05Z

[root@centos7 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           3789        2696         340         193         752         670
Swap:             0           0           0

I hadn't use the docker, if I build without flags like this, will be ok

go build -o hugo main.go

ianlancetaylor · 2020-08-27T03:52:42Z

CC @randall77

I assume the compiler runs out of memory building the large composite literals.

How hard would it be for you to try the recent 1.15 release?

davecheney · 2020-08-27T06:33:56Z

How many cores does this system expose to userspace?

davecheney · 2020-08-27T06:35:39Z

@ianlancetaylor this might have something to do with it https://github.com/yuin/goldmark/blob/master/util/html5entities.go

zhangguanzhang · 2020-08-27T08:23:56Z

CC @randall77

I assume the compiler runs out of memory building the large composite literals.

How hard would it be for you to try the recent 1.15 release?
1.15 run, the same error

# github.com/yuin/goldmark/util
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xc54325, 0x16)
	/usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.sysMap(0xc07c000000, 0x4000000, 0x1160558)
	/usr/local/go/src/runtime/mem_linux.go:169 +0xc6
runtime.(*mheap).sysAlloc(0x1143a80, 0x400000, 0x42b9b7, 0x1143a88)
	/usr/local/go/src/runtime/malloc.go:727 +0x1e5
runtime.(*mheap).grow(0x1143a80, 0x11, 0x0)
	/usr/local/go/src/runtime/mheap.go:1344 +0x85
runtime.(*mheap).allocSpan(0x1143a80, 0x11, 0x7f2c170d0100, 0x1160568, 0x41621b)
	/usr/local/go/src/runtime/mheap.go:1160 +0x6b6
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:907 +0x65
runtime.(*mheap).alloc(0x1143a80, 0x11, 0x7f2c170d0101, 0x7f2c16e96090)
	/usr/local/go/src/runtime/mheap.go:901 +0x85
runtime.largeAlloc(0x21d70, 0xc000010101, 0x7f2c16e96090)
	/usr/local/go/src/runtime/malloc.go:1177 +0x92
runtime.mallocgc.func1()
	/usr/local/go/src/runtime/malloc.go:1071 +0x46
runtime.systemstack(0x0)
	/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1116

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc04f3211a0 sp=0xc04f321198 pc=0x46afe0
runtime.mallocgc(0x21d70, 0xbff940, 0xffffff070001f301, 0xc001670fa0)
	/usr/local/go/src/runtime/malloc.go:1070 +0x938 fp=0xc04f321240 sp=0xc04f3211a0 pc=0x40dc38
runtime.makeslice(0xbff940, 0x0, 0x21d7, 0xc07bfc6000)
	/usr/local/go/src/runtime/slice.go:98 +0x6c fp=0xc04f321270 sp=0xc04f321240 pc=0x44df4c
cmd/compile/internal/ssa.(*regAllocState).computeLive(0xc008112180)
	/usr/local/go/src/cmd/compile/internal/ssa/regalloc.go:2482 +0x1897 fp=0xc04f3215d0 sp=0xc04f321270 pc=0x6aff57
cmd/compile/internal/ssa.(*regAllocState).init(0xc008112180, 0xc00164a000)
	/usr/local/go/src/cmd/compile/internal/ssa/regalloc.go:665 +0x6b9 fp=0xc04f321708 sp=0xc04f3215d0 pc=0x6a0159
cmd/compile/internal/ssa.regalloc(0xc00164a000)
	/usr/local/go/src/cmd/compile/internal/ssa/regalloc.go:145 +0x4a fp=0xc04f321730 sp=0xc04f321708 pc=0x69dc6a
cmd/compile/internal/ssa.Compile(0xc00164a000)
	/usr/local/go/src/cmd/compile/internal/ssa/compile.go:93 +0x9d1 fp=0xc04f3253f8 sp=0xc04f321730 pc=0x64e2f1
cmd/compile/internal/gc.buildssa(0xc00164adc0, 0x0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:460 +0xd25 fp=0xc04f325588 sp=0xc04f3253f8 pc=0xabf085
cmd/compile/internal/gc.compileSSA(0xc00164adc0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:317 +0x5d fp=0xc04f325650 sp=0xc04f325588 pc=0xa882fd
cmd/compile/internal/gc.compile(0xc00164adc0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:275 +0x345 fp=0xc04f3256c8 sp=0xc04f325650 pc=0xa880e5
cmd/compile/internal/gc.funccompile(0xc00164adc0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:220 +0xc5 fp=0xc04f325720 sp=0xc04f3256c8 pc=0xa87c85
cmd/compile/internal/gc.fninit(0xc000fc4480, 0x4b, 0x90)
	/usr/local/go/src/cmd/compile/internal/gc/init.go:63 +0x497 fp=0xc04f325838 sp=0xc04f325720 pc=0xa4ba17
cmd/compile/internal/gc.Main(0xc6e510)
	/usr/local/go/src/cmd/compile/internal/gc/main.go:754 +0x3cc6 fp=0xc04f325f10 sp=0xc04f325838 pc=0xa5f726
main.main()
	/usr/local/go/src/cmd/compile/main.go:52 +0xb1 fp=0xc04f325f88 sp=0xc04f325f10 pc=0xba6471
runtime.main()
	/usr/local/go/src/runtime/proc.go:204 +0x209 fp=0xc04f325fe0 sp=0xc04f325f88 pc=0x439249
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc04f325fe8 sp=0xc04f325fe0 pc=0x46cc21

zhangguanzhang · 2020-08-27T08:25:14Z

How many cores does this system expose to userspace?

[root@centos7 hugo]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            15
Model:                 6
Model name:            Common KVM processor
Stepping:              1
CPU MHz:               3599.998
BogoMIPS:              7199.99
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
L3 cache:              16384K
NUMA node0 CPU(s):     0,1
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology eagerfpu pni cx16 x2apic hypervisor lahf_lm

davecheney · 2020-08-27T08:54:57Z

Yeah, sadly you're probably going to need a bit more physical ram if you're going to run without swap.

zhangguanzhang · 2020-08-27T10:19:18Z

thanks, I change the ram to 8G, it works

[root@centos7 hugo]# free -g
              total        used        free      shared  buff/cache   available
Mem:              7           0           6           0           0           7
Swap:             0           0           0
[root@centos7 hugo]# go install -gcflags='all=-trimpath=/root/hugo -N -l' -asmflags 'all=-trimpath=/root/hugo'

davecheney · 2020-08-27T13:00:29Z

Disabling optimisations probably uses a lot more memory.

If you add just a few kilobytes of swap the operating system will behave differently.

ianlancetaylor changed the title ~~build err "github.com/yuin/goldmark/util fatal error: runtime: out of memory"~~ cmd/compile: runs out of memory building github.com/yuin/goldmark/util Aug 27, 2020

ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 27, 2020

ianlancetaylor added this to the Go1.16 milestone Aug 27, 2020

zhangguanzhang closed this as completed Aug 27, 2020

randall77 mentioned this issue May 1, 2021

cmd/compile: run out of memory during -N -l compilation #45897

Closed

golang locked and limited conversation to collaborators Aug 27, 2021

gopherbot added the FrozenDueToAge label Aug 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/compile: runs out of memory building github.com/yuin/goldmark/util #41058

cmd/compile: runs out of memory building github.com/yuin/goldmark/util #41058

zhangguanzhang commented Aug 27, 2020 •

edited

davecheney commented Aug 27, 2020

zhangguanzhang commented Aug 27, 2020

ianlancetaylor commented Aug 27, 2020

davecheney commented Aug 27, 2020

davecheney commented Aug 27, 2020

zhangguanzhang commented Aug 27, 2020

zhangguanzhang commented Aug 27, 2020

davecheney commented Aug 27, 2020

zhangguanzhang commented Aug 27, 2020

davecheney commented Aug 27, 2020

cmd/compile: runs out of memory building github.com/yuin/goldmark/util #41058

cmd/compile: runs out of memory building github.com/yuin/goldmark/util #41058

Comments

zhangguanzhang commented Aug 27, 2020 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

davecheney commented Aug 27, 2020

zhangguanzhang commented Aug 27, 2020

ianlancetaylor commented Aug 27, 2020

davecheney commented Aug 27, 2020

davecheney commented Aug 27, 2020

zhangguanzhang commented Aug 27, 2020

zhangguanzhang commented Aug 27, 2020

davecheney commented Aug 27, 2020

zhangguanzhang commented Aug 27, 2020

davecheney commented Aug 27, 2020

zhangguanzhang commented Aug 27, 2020 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?