Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/compile: compilation of a few large maps blows up compilers memory usage #22919

Closed
gustafj opened this issue Nov 29, 2017 · 4 comments
Closed

cmd/compile: compilation of a few large maps blows up compilers memory usage #22919

gustafj opened this issue Nov 29, 2017 · 4 comments
Labels
FrozenDueToAge

Comments

@gustafj
Copy link

gustafj commented Nov 29, 2017

What version of Go are you using (go version)?

1.9.2

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GCCGO="gccgo" CC="gcc" CXX="g++" CGO_ENABLED="1" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build186394948=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Compile (go tool compile, go build, go run, etc)

See gist https://gist.github.com/gustafj/aa3c2e7ba4b3dc2b4701282e73d81646 for a sample that generates a sample program that ends up using ~1GB of ram.

  • $ go run main.go
  • $ $(go tool -n compile) -memprofile m.prof -goversion go1.9.2 /tmp/go_compile_test.go
  • $ go tool pprof -top -alloc_space go tool -n compile m.prof | head -20
File: compile
Type: alloc_space
Showing nodes accounting for 1215.06MB, 94.58% of 1284.72MB total
Dropped 125 nodes (cum <= 6.42MB)
      flat  flat%   sum%        cum   cum%
  852.05MB 66.32% 66.32%   852.05MB 66.32%  cmd/compile/internal/ssa.(*regAllocState).computeLive /usr/lib/go/src/cmd/compile/internal/ssa/regalloc.go
   59.02MB  4.59% 70.92%    59.02MB  4.59%  cmd/compile/internal/ssa.(*Func).newValue /usr/lib/go/src/cmd/compile/internal/ssa/func.go
   50.51MB  3.93% 74.85%    50.51MB  3.93%  cmd/compile/internal/gc.(*Progs).Prog /usr/lib/go/src/cmd/compile/internal/gc/gsubr.go
   35.74MB  2.78% 77.63%    35.74MB  2.78%  cmd/compile/internal/ssa.(*Func).setHome /usr/lib/go/src/cmd/compile/internal/ssa/stackalloc.go
      31MB  2.41% 80.04%       31MB  2.41%  cmd/compile/internal/gc.nodl /usr/lib/go/src/cmd/compile/internal/gc/subr.go
   20.21MB  1.57% 81.62%   875.33MB 68.13%  cmd/compile/internal/ssa.(*regAllocState).init /usr/lib/go/src/cmd/compile/internal/ssa/regalloc.go
   15.54MB  1.21% 82.82%    15.54MB  1.21%  cmd/compile/internal/ssa.(*stackAllocState).stackalloc /usr/lib/go/src/cmd/compile/internal/ssa/stackalloc.go
   15.28MB  1.19% 84.01%    15.83MB  1.23%  cmd/compile/internal/ssa.cse /usr/lib/go/src/cmd/compile/internal/ssa/cse.go
   15.26MB  1.19% 85.20%    15.26MB  1.19%  cmd/compile/internal/ssa.storeOrder /usr/lib/go/src/cmd/compile/internal/ssa/schedule.go
   13.17MB  1.03% 86.23%    13.17MB  1.03%  cmd/compile/internal/ssa.schedule /usr/lib/go/src/cmd/compile/internal/ssa/schedule.go
   11.50MB   0.9% 87.12%    11.50MB   0.9%  cmd/compile/internal/gc.newnamel /usr/lib/go/src/cmd/compile/internal/gc/subr.go
    9.10MB  0.71% 87.83%     9.10MB  0.71%  cmd/compile/internal/ssa.(*Func).NewBlock /usr/lib/go/src/cmd/compile/internal/ssa/func.go
    8.72MB  0.68% 88.51%     8.72MB  0.68%  cmd/compile/internal/ssa.(*stackAllocState).buildInterferenceGraph /usr/lib/go/src/cmd/compile/internal/ssa/stackalloc.go
    8.72MB  0.68% 89.19%    20.39MB  1.59%  cmd/compile/internal/ssa.(*stackAllocState).init /usr/lib/go/src/cmd/compile/internal/ssa/stackalloc.go
    8.53MB  0.66% 89.85%     8.53MB  0.66%  cmd/internal/obj/x86.(*AsmBuf).doasm /usr/lib/go/src/cmd/internal/obj/data.go

What did you expect to see?

Fast compilation and a memory usage of max a couple of megs.

What did you see instead?

Slow compilation and a memory usage of almost a gig.
(With a little larger map the end result will be SIGKILL due to out of memory, which for me is ~ 30GB)
@ALTree
Copy link
Member

ALTree commented Nov 29, 2017 •
edited

I think you're hitting #21713. I tried your reproducer on tip and the memory usage seems to be of just a few MB. Can you try with tip (to be 1.10) and confirm that the issue is indeed already fixed?

@ALTree ALTree changed the title Compilation of a few large maps blows up compilers memory usage cmd/compile: compilation of a few large maps blows up compilers memory usage Nov 29, 2017
@gustafj
Copy link
Author

gustafj commented Nov 29, 2017

Thanks for the fast response.
The issue is indeed fixed in tip.
Both my test and the original code that triggered the issue compiles with "normal" memory footprint (2.5MB / 37MB) when using tip.

@ALTree
Copy link
Member

ALTree commented Nov 29, 2017 •
edited

Nice. Since this is fixed on tip (and the fix won't be backported to earlier releases) there's nothing left to do, so I'm closing this issue. The fix will be included in the next go release (1.10).

@ALTree ALTree closed this as completed Nov 29, 2017
@gustafj
Copy link
Author

gustafj commented Nov 29, 2017

Ok, just for reference (since it is discussed in the referenced issue), the code that triggers this behaviour is not generated.
It is a couple quite large lookup tables which are manually created.

@golang golang locked and limited conversation to collaborators Nov 29, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gustafj @ALTree @gopherbot