Can you find out what is being compiled when the compiler uses that much memory?

You wrote that what you're doing is:

bootstrapping go1.12.7 using go14

so I assume you're measuring the memory consumption of the old go1.4 toolchain while it builds the newest toolchain. That old 1.4 toolchain was written in C and it is kept around just for bootstrapping purposes, and it's basically frozen. It won't be optimized to make it faster or consume less memory.

Your screenshot shows the process using 675M of Resident Memory (RES), which is what you care about, so the title of your issue doesn't match your report, as it doesn't appear the toolchain is using 1.7GB of memory.

The newest compilers are written in Go and they also requires a fair amount of memory when bootstrapping master, anyway. You won't be able to compile the Go toolchain using 100 or 500 MB. AFAIK on 64-bit machines you'll need close to 800MB of memory per core. If you restrict make.bash to use one core, you should be able to build the latest toolchain using around 1GB of memory.

@ianlancetaylor I took a snapshot of bootstrapping both go1.10.8 and go1.12.7, and the line with the highest CPU usage appeared when cmd/compile/internal/ssa was compiled:

VSZ: 947624
RSS: 700000
Time: 0:33.66

/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/pkg/tool/netbsd_amd64/compile \
-o \
/tmp/go-build924606097/b082/_pkg_.a \
-trimpath \
/tmp/go-build924606097/b082 \
-p \
cmd/compile/internal/ssa \
-std \
-complete \
-buildid \
U5uFnHi9i-u0_cTbqOyH/U5uFnHi9i-u0_cTbqOyH \
-D \
 \
-importcfg \
/tmp/go-build924606097/b082/importcfg \
-pack \
-c=3 \
/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/src/cmd/compile/internal/ssa/biasedsparsemap.go \
/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/src/cmd/compile/internal/ssa/block.go \
/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/src/cmd/compile/internal/ssa/branchelim.go \
... \
/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/src/cmd/compile/internal/ssa/writebarrier.go \
/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/src/cmd/compile/internal/ssa/zcse.go

I'm impressed that it apparently takes 33 seconds and 700 MB to build this single package (if I'm interpreting the numbers correctly). In a previous build I didn't have swap memory enabled, and the build failed even earlier (the process 6g was killed then). Indeed, this is still less than 1 GB, so I must have measured something else when I initially wrote this issue.

Compared to go1.10.8, the largest process needed 27 MB more, which is a 4% increase. Since this is not much, it may have been pure coincidence that building go1.12 failed for me and building go1.10 had succeeded. Because of that difference I had opened this issue.

I still think 700 MB is quite a lot of memory for building a single go package, but that's probably already mentioned in several other issues, making this one redundant. Therefore I'm closing it.

Thanks for the info. CC'ing @randall77 as FYI, although the issue is now closed.

Yes, cmd/compile/internal/ssa is a very large package. It contains lots of autogenerated code, 375K lines in total. The memory used to compile it is something to keep an eye on, but I don't think we need to do anything at the moment. (Only Go developers need to compile this package, not Go users.) If we need to, it wouldn't be too hard to split out the autogenerated code into separate packages.