Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: Slowdown in compile time on 1.4.1 with cgo #9664

Closed
jscheel opened this issue Jan 22, 2015 · 12 comments
Closed

cmd/go: Slowdown in compile time on 1.4.1 with cgo #9664

jscheel opened this issue Jan 22, 2015 · 12 comments
Labels
FrozenDueToAge
Milestone
Go1.5

Comments

@jscheel
Copy link

jscheel commented Jan 22, 2015

Just upgraded from go1.3.2 darwin/amd64 to go version go1.4.1 darwin/amd64. Simple use of cgo now compiles much slower.

package main

import "C"

func main() {

}
@ianlancetaylor
Copy link
Contributor

Please run "go build -x" with both versions. Please also give some timings: what does "much slower" mean?

@ianlancetaylor ianlancetaylor changed the title Slowdown in compile time on 1.4.1 with cgo cmd/go: Slowdown in compile time on 1.4.1 with cgo Jan 22, 2015
@ianlancetaylor ianlancetaylor added this to the Go1.5 milestone Jan 22, 2015
@minux
Copy link
Member

minux commented Jan 22, 2015

And how did you install 1.4.1? Normally this is caused by incorrect cross
compilation setup.

@jscheel
Copy link
Author

jscheel commented Jan 27, 2015

@ianlancetaylor Here's the difference between 1.3 and 1.4:

jscheel@Jareds-MacBook-Pro:~/go/src/github.com/jscheel/test$ go version
go version go1.3 darwin/amd64
jscheel@Jareds-MacBook-Pro:~/go/src/github.com/jscheel/test$ time go build cgo_benchmark.go

real    0m0.201s
user    0m0.107s
sys 0m0.090s
jscheel@Jareds-MacBook-Pro:~/go/src/github.com/jscheel/test$ gvm use system
Now using version system
jscheel@Jareds-MacBook-Pro:~/go/src/github.com/jscheel/test$ go version
go version go1.4.1 darwin/amd64
jscheel@Jareds-MacBook-Pro:~/go/src/github.com/jscheel/test$ time go build cgo_benchmark.go

real    0m2.791s
user    0m0.334s
sys 0m0.271s
jscheel@Jareds-MacBook-Pro:~/go/src/github.com/jscheel/test$ go build -x cgo_benchmark.go
WORK=/var/folders/27/fqmq58ks6kzbkmmgg7whv4280000gn/T/go-build397696822
mkdir -p $WORK/command-line-arguments/_obj/
cd /Users/jscheel/go/src/github.com/jscheel/test
CGO_LDFLAGS="-g" "-O2" /usr/local/go/pkg/tool/darwin_amd64/cgo -objdir $WORK/command-line-arguments/_obj/ -- -I $WORK/command-line-arguments/_obj/ cgo_benchmark.go
/usr/local/go/pkg/tool/darwin_amd64/6c -F -V -w -trimpath $WORK -I $WORK/command-line-arguments/_obj/ -I /usr/local/go/pkg/darwin_amd64 -o $WORK/command-line-arguments/_obj/_cgo_defun.6 -D GOOS_darwin -D GOARCH_amd64 $WORK/command-line-arguments/_obj/_cgo_defun.c
clang -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common -print-libgcc-file-name
clang -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common -I $WORK/command-line-arguments/_obj/ -g -O2 -o $WORK/command-line-arguments/_obj/_cgo_main.o -c $WORK/command-line-arguments/_obj/_cgo_main.c
clang -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common -I $WORK/command-line-arguments/_obj/ -g -O2 -o $WORK/command-line-arguments/_obj/_cgo_export.o -c $WORK/command-line-arguments/_obj/_cgo_export.c
clang -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common -I $WORK/command-line-arguments/_obj/ -g -O2 -o $WORK/command-line-arguments/_obj/cgo_benchmark.cgo2.o -c $WORK/command-line-arguments/_obj/cgo_benchmark.cgo2.c
clang -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common -o $WORK/command-line-arguments/_obj/_cgo_.o $WORK/command-line-arguments/_obj/_cgo_main.o $WORK/command-line-arguments/_obj/_cgo_export.o $WORK/command-line-arguments/_obj/cgo_benchmark.cgo2.o -g -O2
/usr/local/go/pkg/tool/darwin_amd64/cgo -objdir $WORK/command-line-arguments/_obj/ -dynimport $WORK/command-line-arguments/_obj/_cgo_.o -dynout $WORK/command-line-arguments/_obj/_cgo_import.c
/usr/local/go/pkg/tool/darwin_amd64/6c -F -V -w -trimpath $WORK -I $WORK/command-line-arguments/_obj/ -I /usr/local/go/pkg/darwin_amd64 -o $WORK/command-line-arguments/_obj/_cgo_import.6 -D GOOS_darwin -D GOARCH_amd64 $WORK/command-line-arguments/_obj/_cgo_import.c
clang -I . -fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common -o $WORK/command-line-arguments/_obj/_all.o $WORK/command-line-arguments/_obj/_cgo_export.o $WORK/command-line-arguments/_obj/cgo_benchmark.cgo2.o -g -O2 -Wl,-r -nostdlib
/usr/local/go/pkg/tool/darwin_amd64/6g -o $WORK/command-line-arguments.a -trimpath $WORK -p command-line-arguments -D _/Users/jscheel/go/src/github.com/jscheel/test -I $WORK -pack $WORK/command-line-arguments/_obj/_cgo_gotypes.go $WORK/command-line-arguments/_obj/cgo_benchmark.cgo1.go
pack r $WORK/command-line-arguments.a $WORK/command-line-arguments/_obj/_cgo_import.6 $WORK/command-line-arguments/_obj/_cgo_defun.6 $WORK/command-line-arguments/_obj/_all.o # internal
cd .
/usr/local/go/pkg/tool/darwin_amd64/6l -o cgo_benchmark -L $WORK -extld=clang $WORK/command-line-arguments.a

@minux I originally used the osx binary for both versions. To go back and test, I used gvm to get 1.3.

@minux
Copy link
Member

minux commented Jan 27, 2015

I don't see anything unusual.

Given that go1.4.1 user time + sys time is much less than the real time, I think this might be
a disk cache issue. 1.3 is fast because all needed files are already in RAM, but that's not the
case for 1.4.1.

Both sys time and user time tripled for 1.4.1, as compared to 1.3, I don't know why yet.

My benchmark with warm disk cache doesn't show any significant differences between 1.3.3
and 1.4.1
$ /usr/bin/time go133 build cgohello.go
0.20user 0.05system 0:00.27elapsed 96%CPU (0avgtext+0avgdata 22944maxresident)k
0inputs+3224outputs (0major+24183minor)pagefaults 0swaps
$ /usr/bin/time go141 build cgohello.go
0.19user 0.05system 0:00.25elapsed 94%CPU (0avgtext+0avgdata 23176maxresident)k
16inputs+4112outputs (1major+24846minor)pagefaults 0swaps

@minux
Copy link
Member

minux commented Jan 27, 2015

On OS X with warm cache,

$ /usr/bin/time go133 build cgobench.go
0.59 real 0.33 user 0.20 sys
$ /usr/bin/time go14 build cgobench.go
0.64 real 0.33 user 0.20 sys

Close as not reproducible.

@minux minux closed this as completed Jan 27, 2015
@jscheel
Copy link
Author

jscheel commented Jan 27, 2015

@minux forgive my ignorance, but after compiling once, shouldn't the number speed up the next time? I get the same result every time.

@dvyukov
Copy link
Member

dvyukov commented Jan 27, 2015

You need to install rather than build.

@minux
Copy link
Member

minux commented Jan 27, 2015

So every time the user time + sys time is significantly smaller than real
time, no matter how many times you've run this command? That might be due
to some external factors, e.g. virus scanner, busy machine, etc.

@jscheel
Copy link
Author

jscheel commented Jan 27, 2015

@minux but it's different when I swap go versions, all within the same minute or two. No virus scanner, no real load on the cpu. Very odd. And the slowdown happened immediately after installing 1.4.

@minux
Copy link
Member

minux commented Jan 27, 2015

That is strange.

Can any OS X user reproduce this problem? I can't.

@dooferlad
Copy link

See http://stackoverflow.com/questions/24341654/go-build-became-very-slow-after-installing-a-new-version-of-go

I hit this. Running a simple go test -x showed that go was compiling every library that the test was using. After deleting $GOPATH/pkg and running go install ./... for my project each go test just compiled the test files and picked up pre-compiled libraries.

@jmhodges
Copy link
Contributor

jmhodges commented May 1, 2015

I'm seeing this as well on go test github.com/letsencrypt/boulder/ca if I don't have their copy of mattn/go-sqlite3 installed. Folks on that project came to me and asked me why their builds were slow.

Running go install github.com/letsencrypt/boulder/Godeps/_workspace/src/github.com/mattn/go-sqlite3 corrected the issue, but it seems like it would be useful for go test to handle this for us.

You can repro on sha b4f84714470109201bcbcaa78bcff3c849def0da of letsencrypt/boulder if you wish.

@golang golang locked and limited conversation to collaborators Jun 25, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.5
Development

No branches or pull requests
7 participants
@jmhodges @minux @jscheel @dvyukov @dooferlad @ianlancetaylor @gopherbot