Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: only rebuild dependent packages when export data has changed #15752

Open
josharian opened this issue May 19, 2016 · 10 comments
Open

cmd/go: only rebuild dependent packages when export data has changed #15752

josharian opened this issue May 19, 2016 · 10 comments
Labels
GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done. ToolSpeed
Milestone

Comments

@josharian
Copy link
Contributor

@mwhudson suggested in #15734:

If the export data hasn't changed, can you skip the compilation of dependent packages?

Moving this idea here to be tracked separately.

My 2c: Interesting idea, seems worth pursuing. Not sure how often it happens that code changes don't impact export data (or how cheap that is to detect), but when it does happen, that could save a bunch of computation.

@dr2chase
Copy link
Contributor

It's not trivial. Years a go I worked on an ahead-of-time compiler for Java bytecodes that did just this, and it was a good thing (back in the days of 200Mhz CPUs), but it required substantial amounts of dependence information (stored on disk in gzip format) and it also required that every reference to information from another package log an entry in the internal dependence database for later storage -- in particular, every field reference or interface method reference, because those involve offsets, and offsets might change as interface declarations or field declarations are changed (granularity matters, too -- do you record the actual dependence that field F is at offset 8, or do you just record that the SHA of the struct had a particular value?

@josharian
Copy link
Contributor Author

IIUC, the plan would be to do something like check the SHA of the export data. If that hasn't changed, then the downstream compilation won't change.

@mwhudson
Copy link
Contributor

Yeah, I was just thinking that you'd store the SHA1 of the export data of
all imported packages in the .a file and only recompile if any of them had
changed. No sense trying to be too sophisticated.

I don't know often this would be useful. Certainly sometimes (especially in
a "add more and more debug prints" cycle), but it will obviously also have
a small cost all the time. Seems worth trying out, I'll try to hack
something up between now and 1.8 early.

On 20 May 2016 at 06:56, Josh Bleecher Snyder notifications@github.com
wrote:

IIUC, the plan would be to do something like check the SHA of the export
data. If that hasn't changed, then the downstream compilation won't change.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15752 (comment)

@griesemer
Copy link
Contributor

Régis Crelier (currently at Google) has investigated this issue for Oberon in very much detail - I believe much of his work should be directly applicable to Go. For details see:
http://e-collection.library.ethz.ch/eserv/eth:39386/eth-39386-02.pdf

@josharian
Copy link
Contributor Author

@rsc will this be a part of your cmd/go re-building improvements?

@rsc
Copy link
Contributor

rsc commented Oct 25, 2017

Yes the plan is that we'll be set up well to take advantage of this.
We probably won't do it initially, but as a followup.

@bcmills bcmills added the GoCommand cmd/go label Oct 9, 2018
@josharian
Copy link
Contributor Author

Note that if we do much with #25999, the set of code changes that leave export data unchanged might shrink significantly.

@josharian
Copy link
Contributor Author

cc @bcmills @jayconrod

Is this now just (“just”) a matter of only hashing the export data when determining whether object files are stale?

@bcmills
Copy link
Contributor

bcmills commented Apr 30, 2019

I think so, yes.

@mwhudson
Copy link
Contributor

I think it's a little bit more than that because you still need to re-link any executables?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done. ToolSpeed
Projects
None yet
Development

No branches or pull requests

7 participants