gccgo: build process should not be serial bottlenecked #24962
Comments
|
Eh... perhaps I was premature in opening this. |
|
The way to approach this is to run Note that Go is not C/C++. Although Go often compiles faster, it is more prone to serial bottlenecks. |
|
oops.. well since you commented, seems fair to add a reply... |
|
Whether building with gccgo or the go tool, the Go compilation model has serialization that the C/C++ model does not. In C/C++ a dependency is expressed by #include a .h file. In Go a dependency is expressed by importing a package. In C/C++ the .h file is already present in the source tree. In Go the imported package must be compiled. The effect is that C/C++ compilation does much more work overall: the .h file is read and parsed many times. But on the other hand the C/C++ compilation is more parallelizable, in that you can compile code immediately without having to wait for the dependencies to be compiled. |
|
I think you are saying that using "the go compilation model", you may wish
to compile file A, but that depends on file B, which also depends on C and
D, which first need to be compiled. so there is blocking involved.
okay i get that.
however, unless there is a VERY deep dependency chain that includes
practically every file, that shouldn't be a problem, for properly ordered
compile order.
if you have effectively unlimited cpus for parallelism, then in theory, the
entire set should be compilable in a number of compile cycles equal to the
deepest dependancy chain.
That is, lets say that files (A,B,C,D) all depend on some mix of (E,F,G,H).
And they all depend on some mix of (I,J,K L). and THEY all depend on (M,N,O
P)
That is a tree of depth 4. If you have unlimited cpus, then the whole lot
should theoretically be compilable in 4 rounds of parallel compiles.
I have 100 cpus. So, in theory, I should be able to compile the libgo part
of things very fast, with "gmake -j".
Even if there are multiple bottlenecks of (oh we need to group these sets
of files into libraryX.a first), it should be possible to paralellize those
builds.
So while the system is busy building library libfmt.a, it should ALSO be
building libflag.a, libcompress.a, etc. at the same time.
and yes I get that some of those libraries depend on other libraries :)
But you see my point. The library builds should be parallelizable as well,
if library dependancies are first mapped out.
Apparently, there are 169 library type files, judging by
$ find . -name '*.dep' |wc -l
169
That is equivalent to the PACKAGES definition in the Makefile, I believe.
But I'm having difficult finding the build trigger for PACKAGES in the
Makefile, and whether it is parallel or serial.
…On Friday, April 20, 2018, Ian Lance Taylor ***@***.***> wrote:
Whether building with gccgo or the go tool, the Go compilation model has
serialization that the C/C++ model does not. In C/C++ a dependency is
expressed by #include a .h file. In Go a dependency is expressed by
importing a package. In C/C++ the .h file is already present in the source
tree. In Go the imported package must be compiled. The effect is that C/C++
compilation does much more work overall: the .h file is read and parsed
many times. But on the other hand the C/C++ compilation is more
parallelizable, in that you can compile code immediately without having to
wait for the dependencies to be compiled.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#24962 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABpK-SDGxHaPUaErSi-2BGpyHCQbQ8EMks5tqeURgaJpZM4Tc1qm>
.
|
|
The libgo build is parallelizable and it is parallelized. But you still hit a roadblock. In my experience the problem is the net/http package, which is one of the most complex packages and also has a great deal of dependencies. So you have to build most of libgo, and only then do you get to build net/http, and then you block until that specific build is complete, and then you build in parallel a few remaining packages that depend on net/http. The build trigger for |
this is for the gcc 7.3.0 codebase.
I have a 100 cpu-thread box.
I'm running with gmake -j 50
It hits some part of the go build stage.. and then grinds to a crawl.
here is a snippet of some of the crazy long output at this point:
f=
echo net/smtp.lo | sed -e 's/.lo$/.o/'; gobjcopy -j .go_export $f net/smtp.s-gox.tmp; /bin/sh ../../../../libgo/mvifdiff.sh net/smtp.s-gox.tmpecho net/smtp.s-gox | sed -e 's/s-gox/gox/'echo timestamp > net/smtp.s-gox
sorry I'm not a GNU makefile expert, so cant suggest a patch :(
I just know that when it builds normal C or C++ based libs, it does the nice massively parallel compiles, but when it hits that stage of compling, with
libtool (blahblah) gccgo blahblah..
suddenly, it seems to stop parallelizing for a long time, comparatively speaking
The text was updated successfully, but these errors were encountered: