Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/vgo: lookup/fetch dependencies concurrently #24316

Closed
awly opened this issue Mar 8, 2018 · 7 comments
Closed

x/vgo: lookup/fetch dependencies concurrently #24316

awly opened this issue Mar 8, 2018 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@awly
Copy link
Contributor

awly commented Mar 8, 2018

Ran vgo build for program that imports a large dependency tree (imports k8s.io/kubernetes/something).

It worked for several minutes, printing lines like this at a steady pace:
vgo: finding github.com/kr/pretty v0.0.0-20140723054909-088c856450c0

It looks like vgo works through list dependencies sequentially (recursively appending additional ones as they are discovered). Perhaps this could be parallellized?

@gopherbot gopherbot added this to the vgo milestone Mar 8, 2018
@awly
Copy link
Contributor Author

awly commented Mar 9, 2018 via email

@awly
Copy link
Contributor Author

awly commented Mar 9, 2018

Hmm, got an email about someone commenting but perhaps they deleted the comment.
Anyway, the above description still stands.

@AlexRouSg
Copy link
Contributor

Sorry about deleting it, shortly after posting I realized roughly the same thing you posted.

Also based on https://research.swtch.com/vgo-module
It defines a url to get the go.mod file
GET baseURL/module/@v/version.mod fetches the go.mod file for that version.

So maybe it could (not sure if it does already) just get all the mod files and build the dependency tree and then fetch all the source files completely in parallel. But yeah this wouldn't work with legacy packages so it have to revert to getting the entire package to figure out the dependency tree.

@ALTree
Copy link
Member

ALTree commented Mar 9, 2018

Same request but for go get for reference: #11821.

@rsc
Copy link
Contributor

rsc commented Mar 27, 2018

Once we are happy with the rest of the code, yes, we should parallelize it. But I want to figure out what the code should do and make it correct before we make it fast.

@rsc rsc added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed NeedsFixButNotYet labels Apr 9, 2018
@rsc
Copy link
Contributor

rsc commented Apr 9, 2018

Brad asked me to kill uses of NeedsFixButNotYet, so I'm going to call this NeedsInvestigation.

@gopherbot
Copy link

Change https://golang.org/cl/119055 mentions this issue: cmd/go: parallelize module load operations

@golang golang locked and limited conversation to collaborators Jun 15, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants