Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: Go compiler eats all available system memory #20714

Closed
zlepper opened this issue Jun 17, 2017 · 10 comments
Closed

cmd/compile: Go compiler eats all available system memory #20714

zlepper opened this issue Jun 17, 2017 · 10 comments
Labels
FrozenDueToAge ToolSpeed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@zlepper
Copy link

zlepper commented Jun 17, 2017

What version of Go are you using (go version)?

go version go1.8 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=E:\development\goprojects
set GORACE=
set GOROOT=D:\Go
set GOTOOLDIR=D:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\hanse\AppData\Local\Temp\go-build350759391=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

What did you do?

Tried to compile to following using go build .
gocompiler.zip

I suspect it's the autogenerated hosting.go file that causes things to explode due to some quite huge string (It's an embedded angular application)

What did you expect to see?

A build program

What did you see instead?

All my memory being eaten.

I have to kill the compiler as it after a couple of seconds uses more than 10GB.
If i just let it run it eats all the memory available on my machine, and the machine has to be reset using the hardware switch.

@bradfitz
Copy link
Contributor

Many of these sorts of bugs were fixed recently.

Could you try Go 1.9beta1? https://golang.org/dl/#unstable

Note that on Windows, the MSI installer will uninstall your old version (that is #16305), so you'll want to use the zip version for testing probably.

@bradfitz bradfitz added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 17, 2017
@bradfitz bradfitz changed the title Go compiler eats all available system memory cmd/compile: Go compiler eats all available system memory Jun 17, 2017
@bradfitz bradfitz added this to the Go1.10 milestone Jun 17, 2017
@zlepper
Copy link
Author

zlepper commented Jun 17, 2017

Attempting to run the build command just gives me the following:

C:\Users\hanse\Desktop\gocompiler>"D:\Downloads\go1.9beta1.windows-amd64\go\bin\go.exe" build .
# _/C_/Users/hanse/Desktop/gocompiler
flag provided but not defined: -goversion
usage: compile [options] file.go...

@bradfitz
Copy link
Contributor

That means your environment is messed up. You're running a mix of Go versions. Unset or fix your GOROOT.

@zlepper
Copy link
Author

zlepper commented Jun 17, 2017

Fixed my env, attempted to compile with 1.9beta1: Memory consumption still explodes.

go version:
go version go1.9beta1 windows/amd64

go env:
set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=E:\development\goprojects
set GORACE=
set GOROOT=D:\Downloads\go1.9beta1.windows-amd64\go
set GOTOOLDIR=D:\Downloads\go1.9beta1.windows-amd64\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\hanse\AppData\Local\Temp\go-build564923191=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

@josharian
Copy link
Contributor

I strongly suspect that this is a duplicate of #16394. If so, it is a real bug that needs fixing, but there is a good workaround available: Avoid string concatenation of many/large constant strings in generated code. Instead, use strconv.Quote to escape the string and emit it verbatim.

@zlepper
Copy link
Author

zlepper commented Jun 18, 2017

Indeed, using strconv.Quote fixed gets arround the issue. (And for what i need, it's much more elegant too :) )

The only difference i see between this issue, and the one @josharian linked, is that the go compiler doesn't actually crash here, it makes the system run out of memory (At least on my system running with 16GB spare)

@josharian
Copy link
Contributor

Given enough memory, the compiler would crash. :) I'm going to call this a dup, since the underlying cause is the same. Thanks for checking.

@zlepper
Copy link
Author

zlepper commented Jun 19, 2017

Shouldn't the compiler give up before it kills the system it's running on? Having to hardware reset the computer is not exactly optimal?

@bradfitz
Copy link
Contributor

bradfitz commented Jun 19, 2017

Shouldn't the compiler give up before it kills the system it's running on? Having to hardware reset the computer is not exactly optimal?

@zlepper, Josh isn't denying there's a problem. He closed it as a dup of #16394.

Go should use as little memory as possible and not go into runaway memory situations like this, but on the other hand, it's not Go's job to tiptoe around your operating system. If your operating system doesn't have the memory, it's the OS's job to kill the process, not wedge itself and require a physical hardware reset.

But this will be fixed. Subscribe to #16394.

@zlepper
Copy link
Author

zlepper commented Jun 19, 2017

That is true, had not considered that angle. Thank you.

@golang golang locked and limited conversation to collaborators Jun 19, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge ToolSpeed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

4 participants