Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: parallelize go get downloads #11821

Closed
jacobsa opened this issue Jul 22, 2015 · 4 comments
Closed

cmd/go: parallelize go get downloads #11821

jacobsa opened this issue Jul 22, 2015 · 4 comments

Comments

@jacobsa
Copy link
Contributor

jacobsa commented Jul 22, 2015

Running go get -u for an app with a decent number of dependencies can be extremely slow from a location with high latency to GitHub (and other code hosts, too -- it's even worse when there are redirects involved). For example, from my machine in Sydney the following command takes almost 90 seconds to download 18 repos:

go get -v -u -t github.com/googlecloudplatform/gcsfuse/...

This is in stark contrast to general use of the go command, which is otherwise pleasantly fast.

I notice that downloads happen serially. It seems like go get could benefit greatly from doing them concurrently. Are there any major obstacles to doing so?

@rsc
Copy link
Contributor

rsc commented Jul 22, 2015

The main obstacle is that you have to fetch A to find out that it imports B and C.
But we could parallelize B and C.

@rsc rsc changed the title cmd/go: serial downloads can be very slow cmd/go: parallelize go get downloads Jul 22, 2015
@rsc rsc added this to the Go1.6Early milestone Jul 22, 2015
@nightlyone
Copy link
Contributor

There are also rate-limit issues.

e.g. common setup is to use a shared ssh session to github (sort of a connection pool via ControlPersist yes option) and there are only 10 concurrent connections allowed over that one (hard coded limit in openssh deamon).

So this feature would need a bunch of tuneables.

A different approach to target the increasing complexity of go get and vastly different usage profiles might be a git like plugin architecture for the whole vcs part.

@rsc rsc modified the milestones: Unplanned, Go1.6Early Nov 4, 2015
@powerman
Copy link

Is this correct way (in case GOPATH contains exactly one element) to speedup go get -v -t -u all?

find $GOPATH/src -type d -name .git -printf "%h\0" |
    xargs -0 -P64 -I{} sh -c "cd {}; git pull; git submodule update"
go get -v -t all

I suppose it may miss some corner cases like default branch change on GitHub, but I'm not sure is go get -u handle such cases too.

@bcmills
Copy link
Contributor

bcmills commented Jan 23, 2019

Module downloads are performed in parallel. I don't think that retrofitting GOPATH mode to do so would be worth the effort.

@bcmills bcmills closed this as completed Jan 23, 2019
@golang golang locked and limited conversation to collaborators Jan 23, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

6 participants