Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: add a way to obtain a package's build ID via list -json #37281

Closed
mvdan opened this issue Feb 18, 2020 · 3 comments
Closed

cmd/go: add a way to obtain a package's build ID via list -json #37281

mvdan opened this issue Feb 18, 2020 · 3 comments
Labels
FeatureRequest FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@mvdan
Copy link
Member

mvdan commented Feb 18, 2020

The toolchain computes build IDs of packages from their build input - which I believe includes a package's source code, plus the build IDs of all their dependencies.

This can be seen if one runs go build -x, as compiler invocations contain -buildid $BUILD_ID. Similarly, a package's output file contains such build ID as well, so we can obtain it via these two commands:

$ go list -f {{.Export}} -export os
/home/mvdan/tip/pkg/linux_amd64/os.a
$ go tool buildid $(go list -f {{.Export}} -export os)
WCQPVNh2igTNOQyg4V2a/aDZysAb7fmq4QDTG8hDz

This is rather unfortunate for tools that need to look at thousands of packages at a time, though. One could use the build ID to see if a package hasn't had any changes, and thus avoid extra work. Even if we have thousands of packages, we can load them all at once with one single go list -json call - however, we will need one extra exec call to go tool buildid per package, whose cost can add up quickly.

I propose that we add a new field to go list -json to expose said build ID. For example, BuildID string. It would be fine if populating this field required a flag like -export.

I also realise that build IDs are an internal toolchain detail, which should only matter to compiler and toolchain authors. However, there are plenty of third-party tools out there which are designed to be closely coupled with the toolchain or compiler. For example, tools designed to run via go build -toolexec=mytool can use build IDs to avoid work, or to produce deterministic output based on the input. This is exactly the kind of thing I do for a code obfuscator, only with lots of go tool buildid calls.

I also am not asking for the build ID format to be documented or stabilized. A tool shouldn't interpret the string as more than a hash if it wants to remain compatible with past and future Go versions. Or, if it really wants to parse the hash, it should only do so with very specific versions of the Go tool.

/cc @dominikh

@jayconrod jayconrod added FeatureRequest NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Feb 18, 2020
@jayconrod jayconrod added this to the Backlog milestone Feb 18, 2020
@jayconrod
Copy link
Contributor

cc @bcmills @matloob

@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 13, 2020
@gopherbot gopherbot removed the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label May 13, 2020
@bcmills
Copy link
Contributor

bcmills commented May 13, 2020

Let's see how invasive the change is, and how clean the workflow ends up (as demonstrated in an integration test).

@mvdan mvdan self-assigned this May 13, 2020
@gopherbot
Copy link

Change https://golang.org/cl/263542 mentions this issue: cmd/go: add BuildID to list -json -export

@golang golang locked and limited conversation to collaborators Oct 20, 2021
@rsc rsc unassigned mvdan Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FeatureRequest FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants