Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/compile: compiler hangs without optimizations #29144

Open
csnitker opened this issue Dec 7, 2018 · 10 comments
Open

cmd/compile: compiler hangs without optimizations #29144

csnitker opened this issue Dec 7, 2018 · 10 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed
Milestone
Backlog

Comments

@csnitker
Copy link

csnitker commented Dec 7, 2018

What version of Go are you using (go version)?

$ go version
go version go1.11.2 windows/amd64

Does this issue reproduce with the latest release?

Yes, reproduced using go 1.11, 1.11.2 and 1.10

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\csnitker\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\csnitker\go
set GOPROXY=
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\csnitker\go\src\go.registrar.int.godaddy.com\RegistrarServices\whois\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\csnitker\AppData\Local\Temp\go-build373743682=/tmp/go-build -gno-record-gcc-switches

What did you do?

Run go build -gcflags=all="-N -l" -x main.go or dlv debug main.go on the following:

package main

import "github.com/zonedb/zonedb"

func main() {
    zone := zonedb.PublicZone("godaddy.com")
    if zone != nil {
        println(zone)
    }
}

What did you expect to see?

A successful build.

What did you see instead?

The compiler hangs and consumes large amounts of memory.

image

The command from the process above shows:

C:\go1.11\pkg\tool\windows_amd64\compile.exe -o C:\Users\csnitker\AppData\Local\Temp\go-build516058638\b002\_pkg_.a -trimpath C:\Users\csnitker\AppData\Local\Temp\go-build516058638\b002 -N -l -p github.com/zonedb/zonedb -complete -buildid mw6Bk1kcUZg16C0mq6E3/mw6Bk1kcUZg16C0mq6E3 -goversion go1.11 -D "" -importcfg C:\Users\csnitker\AppData\Local\Temp\go-build516058638\b002\importcfg -pack -c=4 C:\Users\csnitker\go\pkg\mod\github.com\zonedb\zonedb@v0.0.0-20181207082023-cee273e3e759\zone.go C:\Users\csnitker\go\pkg\mod\github.com\zonedb\zonedb@v0.0.0-20181207082023-cee273e3e759\zones.go

It's also interesting that go build -gcflags="-N -l" main.go completes successfully and works using dlv exec main.exe but does not with the commands above. I'm not entirely sure if this is a go problem or an issue with the library (or delve) but figured I would start here since an optimized build completes without issue.
@ianlancetaylor ianlancetaylor changed the title compile hangs without optimizations cmd/compile: compiler hangs without optimizations Dec 7, 2018
@ianlancetaylor
Copy link
Contributor

The file github.com/zonedb/zonedb/zones.go has a fairly large composite literal. I suspect that the compiler isn't hanging as such, it's just taking a lot of memory, perhaps causing your system to thrash.

CC @randall77

@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 7, 2018
@ianlancetaylor ianlancetaylor added this to the Go1.12 milestone Dec 7, 2018
@ianlancetaylor
Copy link
Contributor

Although actually it doesn't take very long to build that package on my laptop.

@randall77
Copy link
Contributor

It's the difference between -gcflags=all="-N -l" and -gcflags="-N -l". The former applies the optimizations-off flags to all packages (including the zonedb package) and the latter to only the top-level package and not its imports. The latter will use an optimized version of the zonedb package which compiles somewhat quickly.

It's the -N, removing -l doesn't change anything.

@randall77
Copy link
Contributor

Looks like it is spending most of its time (and space) in live variable analysis in the register allocator.
I'm not sure there's a whole lot we can do about that, unfortunately.
Maybe if there was a key phase that, if enabled even with -N, would make the function easier for regalloc to handle.

@ianlancetaylor
Copy link
Contributor

That's impressive. With -gcflags=all=-N I let the compiler run for ten minutes before killing it, at which point the virtual memory size was over 15G.

Why do we need live variable analysis with -N? In a big function can we just dump everything on the stack?

@randall77
Copy link
Contributor

We could implement a second, simpler register allocator for -N, sure. I was hoping to avoid that.
I'm not sure how much we could share between the two allocators. The live variable analysis is pretty integral to the current design, I don't think we could just say "everything is live" and have it work.

@andybons andybons modified the milestones: Go1.12, Go1.13 Feb 12, 2019
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@odeke-em odeke-em mentioned this issue Nov 29, 2019
Closed
@icamys
Copy link

icamys commented Nov 29, 2019

Hi all! The bug also can be reproduced with go version go1.13 linux/amd64

@lzaldivarkt
Copy link

Another confirmation. This gets triggered when running go code in intellij on debug mode, if the zonedb package is added to the code, as Intellij adds those gcflags by default.

Version: go version go1.13.3 linux/amd64

@josharian
Copy link
Contributor

If someone wants to dig into this, it might be useful to know whether there is one key optimization omitted by -N that makes the difference. If so, perhaps (say) there might be a way to do a subset of that optimization even with -N to keep sizes reasonable.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022
@ivanmemruk
Copy link

ivanmemruk commented Apr 27, 2023 •
edited

This reproduces on 1.20.3 on darwin/arm64. Specifically, because of this handy module https://github.com/pioz/faker/ which contains some large literals as well, for example the one in https://github.com/pioz/faker/blob/master/address.go (beware, it kinda hangs Chrome for a bit as well).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed
Projects
Go Compiler / Runtime
Status: Triage Backlog
Milestone
Backlog
Development

No branches or pull requests
10 participants
@josharian @rsc @andybons @ivanmemruk @ianlancetaylor @csnitker @randall77 @icamys @gopherbot @lzaldivarkt