Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: compileSSA takes a long time on very large (generated) functions #30077

Open
udhos opened this issue Feb 4, 2019 · 10 comments
Open
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed
Milestone

Comments

@udhos
Copy link

udhos commented Feb 4, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12beta2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/esmarques/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/esmarques/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build549978624=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ go build a.go

a.go is machine-genrated Go code:

a.zip

$ wc a.go
  44611  190970 1405515 a.go

What did you expect to see?

go build should finish eventually.

What did you see instead?

go build consuming all the CPU and never finishes.

@udhos
Copy link
Author

udhos commented Feb 4, 2019

Correction, 'go build' took many minutes but did finish eventually.

@bcmills
Copy link
Contributor

bcmills commented Feb 4, 2019

.zip files are a real pain to use for diagnostics. Can you reproduce the problem in the Go Playground (https://play.golang.org)? If so, please provide a Playground link; otherwise, a link to a public repo would be acceptable.

You say that the build consumes “all the CPU” — exactly which process is it that is stuck? (Is it the compiler, the linker, or something else?)

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Feb 4, 2019
@bcmills bcmills changed the title go build runs forever cmd/go: 'go build' takes a long time on generated source file Feb 4, 2019
@bcmills bcmills added this to the Go1.13 milestone Feb 4, 2019
@bcmills
Copy link
Contributor

bcmills commented Feb 4, 2019

Finally, why is this particular generated file important, and how much control do you have over the generator?

It's almost always possible to build an adversarial input to a compiler, so sometimes it's easier to fix the generator to avoid pathological cases than to fix the compiler to handle them efficiently.

@nussjustin
Copy link
Contributor

nussjustin commented Feb 4, 2019

Building the SSA seems to be the culprit.

Flamegraph from 1 minute of perf recording go tool compile:

image

Edit: go version devel +3fc276ccf8 Mon Feb 4 06:53:49 2019 +0000 linux/amd64

@udhos
Copy link
Author

udhos commented Feb 4, 2019

This is the process consuming CPU:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
32420 esmarqu+  20   0 1769892 1,593g  10620 S 100,0  5,1   0:56.51 compile

Finally, why is this particular generated file important, and how much control do you have over the generator?

Well, I control the code generator. If I find what is the pathological case for the Go compiler, I can avoid it.

Not sure if it should be simple to paste 44k line Go file to Playground, but I am failing at it.

@mvdan
Copy link
Member

mvdan commented Feb 4, 2019

Are you generating very large functions by any chance? I've seen situations where that makes SSA terribly slow, since you essentially create massive scopes for it to take care of. If that's the case, you could try splitting large functions into pieces, and calling all of those.

@udhos
Copy link
Author

udhos commented Feb 4, 2019

Are you generating very large functions by any chance? I've seen situations where that makes SSA terribly slow, since you essentially create massive scopes for it to take care of. If that's the case, you could try splitting large functions into pieces, and calling all of those.

Yes, main() is one single huge function.

@ianlancetaylor
Copy link
Contributor

CC @randall77

@bcmills bcmills changed the title cmd/go: 'go build' takes a long time on generated source file cmd/compile: compileSSA takes a long time on very large (generated) functions Feb 4, 2019
@bcmills
Copy link
Contributor

bcmills commented Feb 4, 2019

See previously #14775, #14774, #17926.

@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Feb 4, 2019
@bcmills bcmills modified the milestones: Go1.13, Unplanned Feb 4, 2019
@udhos
Copy link
Author

udhos commented Feb 4, 2019

The offending code has been located, I guess.

Generated code had multiple large 'switch' statements like this:

switch index {
case 0: goto gosub_return_0
// (snip)
case 111: goto gosub_return_111
}

I reworked code generation to issue a single large 'switch' statment, and jump into it as needed.

Compilation is much faster now.

@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed
Projects
None yet
Development

No branches or pull requests

5 participants