Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/compile: cgo based vendored libraries take forever to run #14081

Closed
pyrossh opened this issue Jan 24, 2016 · 4 comments
Closed

cmd/compile: cgo based vendored libraries take forever to run #14081

pyrossh opened this issue Jan 24, 2016 · 4 comments
Labels
FrozenDueToAge
Milestone
Go1.7

Comments

@pyrossh
Copy link

pyrossh commented Jan 24, 2016

It seems if I install a cgo based library into my vendor packages and try to run my main file it takes a very long time to run but if i don't vendor these packages it runs in normal time.
This is the code I tried to run

package main

import (
    "database/sql"
    _ "github.com/mattn/go-sqlite3"
)
var msDB *sql.DB
func main() {
   println("StartingDB")
}

My GoENV

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/pyros2097/Code"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

I just run go run main.go maybe its an issue with the go compiler I guess it doesn't compile cgo based libraries in vendor packages and put them in this folder pkg/linux_amd64 and I think it might be recompiling them everytime just running the sqlite example takes a minute if its vendored.
The sqlite3 package is present in my vendor/github.com/mattn/go-sqlite3 directory.

I've posted the issue here also Masterminds/glide#223
@ianlancetaylor
Copy link
Contributor

When you say it takes a long time to run, are you measuring the time of go run or the time of running the program by itself after using go install or go build?

If you are measuring the time of go run please look at the output of go run -x to see what it is doing. Try using go run -i and see if that speeds up future invocations.

@ianlancetaylor ianlancetaylor added this to the Go1.7 milestone Jan 24, 2016
@davecheney
Copy link
Contributor

Don't use go run, or go build as they will not cache the result of
compiling your sqlite dependency.

Use go install to build and cache the results of compilation which will
save you a lot of time.

On Mon, 25 Jan 2016, 03:03 Ian Lance Taylor notifications@github.com
wrote:

When you say it takes a long time to run, are you measuring the time of go
run or the time of running the program by itself after using go install
or go build?

If you are measuring the time of go run please look at the output of go
run -x to see what it is doing. Try using go run -i and see if that
speeds up future invocations.

—
Reply to this email directly or view it on GitHub
#14081 (comment).

@pyrossh
Copy link
Author

pyrossh commented Jan 25, 2016

Timing it go run takes this much time

StartingDB

real    0m45.321s
user    0m45.153s
sys 0m0.623s

Yeah when I run it with the -x option it shows that is calling gcc every single time, Something like this

/github.com/mattn/go-sqlite3/_obj/_cgo_main.o -c $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/_cgo_main.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/ -g -O2 -std=gnu99 -DSQLITE_ENABLE_RTREE -DSQLITE_THREADSAFE -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4_UNICODE61 -I. -o $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/_cgo_export.o -c $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/_cgo_export.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/ -g -O2 -std=gnu99 -DSQLITE_ENABLE_RTREE -DSQLITE_THREADSAFE -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4_UNICODE61 -I. -o $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/backup.cgo2.o -c $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/backup.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/ -g -O2 -std=gnu99 -DSQLITE_ENABLE_RTREE -DSQLITE_THREADSAFE -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4_UNICODE61 -I. -o $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/callback.cgo2.o -c $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/callback.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/ -g -O2 -std=gnu99 -DSQLITE_ENABLE_RTREE -DSQLITE_THREADSAFE -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4_UNICODE61 -I. -o $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/error.cgo2.o -c $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/error.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/ -g -O2 -std=gnu99 -DSQLITE_ENABLE_RTREE -DSQLITE_THREADSAFE -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4_UNICODE61 -I. -o $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/sqlite3.cgo2.o -c $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/sqlite3.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/ -g -O2 -std=gnu99 -DSQLITE_ENABLE_RTREE -DSQLITE_THREADSAFE -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4_UNICODE61 -I. -o $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/sqlite3_load_extension.cgo2.o -c $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/sqlite3_load_extension.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/ -g -O2 -std=gnu99 -DSQLITE_ENABLE_RTREE -DSQLITE_THREADSAFE -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4_UNICODE61 -I. -o $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/sqlite3_other.cgo2.o -c $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/sqlite3_other.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/ -g -O2 -std=gnu99 -DSQLITE_ENABLE_RTREE -DSQLITE_THREADSAFE -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4_UNICODE61 -I. -o $WORK/playlyfe.com/potential-connectors/vendor/github.com/mattn/go-sqlite3/_obj/sqlite3-binding.o -c ./sqlite3-binding.c

I'm using go version go1.5.2 linux/amd64 so there is no i option so I could not try it out.

So I tried go install

GOBIN=`pwd` go install main.go

and the first compilation too time but after that it cached it and it became fast
go run main.go

StartingDB 22 22

real    0m0.792s
user    0m0.980s
sys 0m0.090s

@davecheney
Copy link
Contributor

I'm closing this question as answered. We don't use the issue tracker for questions. Please see https://golang.org/wiki/Questions . Thanks.

@golang golang locked and limited conversation to collaborators Jan 24, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.7
Development

No branches or pull requests
4 participants
@davecheney @pyrossh @ianlancetaylor @gopherbot