x/vgo: lookup/fetch dependencies concurrently #24316
Comments
|
For example A depends on B, C, D, E, F, G. D and F depend on X 1.1 and X
1.5 respectively.
Use worker pool of size 3. All "finalized" versions are kept in some
synchronized structure (or map)
- process A.
- process B, C, D, add X 1.1 to queue
- process E, F, G, add X 1.5 to queue
- process X 1.1, X1.5
- if 1.1 gets the lock first, 1.5 overrides the picked version after
- if 1.5 gets the lock first, 1.1 is ignored
If you have "blacklisted" versions, those are recorded in the store and get
handled if the dependency comes up again.
In my experience, many packages have >>2 dependencies and the transient
dependencies don't overlap too often.
Of course, that's from a limited sample size.
The implementation would certainly have more edge cases than serial
approach.
But so far I don't think it's too complex and would benefit some larger
projects.
…On Thu, Mar 8, 2018 at 5:45 PM Chotepud Teo ***@***.***> wrote:
How would you propose it process dependencies in parallel?
If A depends on B and C 1.2 and B depends on C 1.4
There is no way to know to use C 1.4 until it got to B and it won't know
to use B until it got to A.
/cc @rsc <https://github.com/rsc>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#24316 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABF9l3mwE0xYs09JToC1NR6hrH7JwO_4ks5tcd66gaJpZM4SjgTw>
.
|
|
Hmm, got an email about someone commenting but perhaps they deleted the comment. |
|
Sorry about deleting it, shortly after posting I realized roughly the same thing you posted. Also based on https://research.swtch.com/vgo-module So maybe it could (not sure if it does already) just get all the mod files and build the dependency tree and then fetch all the source files completely in parallel. But yeah this wouldn't work with legacy packages so it have to revert to getting the entire package to figure out the dependency tree. |
|
Same request but for |
|
Once we are happy with the rest of the code, yes, we should parallelize it. But I want to figure out what the code should do and make it correct before we make it fast. |
rsc
added
NeedsInvestigation
|
Brad asked me to kill uses of NeedsFixButNotYet, so I'm going to call this NeedsInvestigation. |
|
Change https://golang.org/cl/119055 mentions this issue: |
Ran vgo build for program that imports a large dependency tree (imports k8s.io/kubernetes/something).
It worked for several minutes, printing lines like this at a steady pace:
vgo: finding github.com/kr/pretty v0.0.0-20140723054909-088c856450c0
It looks like vgo works through list dependencies sequentially (recursively appending additional ones as they are discovered). Perhaps this could be parallellized?
The text was updated successfully, but these errors were encountered: