Summary

We have go/packages as a high-level API for loading Go packages. It's abstracted away from go list -json, because it's also meant to support other build systems like Bazel.

I propose that we add a lower-level, simple Go API to call go list -json and go list -json -m. Given that it's only meant to support go list, it can show more information, and it can support listing modules too.

Description

There are plenty of use cases where one can rightfully only support Go modules and GOPATH, and not care about other build systems. For example, a code generation tool that's only used by a project's developers, when they are happy with using modules alone. Or a piece of tooling internal to a company, if they are confident that they won't switch build systems anytime soon.

In those scenarios, using go/packages can be enough if one's needs are simple. For example, looking at the Go files contained in a number of packages. But, in the context of modules, it can be pretty easy to run into limitations for little reason, such as:

• What module does this package belong to?
• Where is this package (in what directory)?
• Is this package part of the standard library?
• Which input patterns matched each resulting package?

All of these are contained by go list -json, but aren't exposed by go/packages, since they aren't common with other Go build systems. That makes sense, but it's a tradeoff that isn't right sometimes.

One can call go list -json directly - for example, to obtain a package's directory alone. Given the structs that one can see in go help list:

out, err := exec.Command("go", "list", "-f", "{{.Dir}}", pkgPath)
if err != nil { ... }
dir := string(bytes.TrimSpace(out))

However, it quickly gets complicated if one needs multiple values at once, or if one wants to look at multiple packages in a single call. For example, these open questions aren't easy to answer, for anyone who isn't very familiar with the Go tool:

• Should I just copy the structs from go help list? How can I update them in the future without manual steps?
• How should I deal with errors? Do I want to use the -e flag?
• If I use -e, can the Go command still return a non-zero status code?

It's possible to make some assumptions or guesses here which might be OK for the user, but even then, the user will likely end up manually copying some code from go help list, and writing 20 to 40 lines of extra code with os/exec and encoding/json.

Proposed solution

As explained before, I propose that we add a package with two new APIs - loading packages, and loading modules. The latter would only work in module mode, just like go list -m. I have a proof of concept implementation here, which I'm using in a couple of projects, and which I'd be happy to donate via a CL.

The code isn't terribly complex or large, but I think it's easy to tell that exposing good errors isn't trivial.

I think x/exp would be a reasonable place for it to live for now, to give us some space to redesign the API later if we want. I haven't thought too hard about the package name itself, but one could imagine list following go list, or modules, as an analogous to packages given that long-term GOPATH will probably be deprecated/removed in favour of modules.

Considered alternatives

1. I seem to recall some ideas about splitting cmd/go into smaller packages, such as what one can find in https://godoc.org/golang.org/x/mod. I think that's a great initiative, and I hope that module is expanded. However, I struggle to imagine how we could ever fit an API like this in x/mod:

• Would we still call go list? The rest of the module is "pure Go" without exec, as far as I can tell
• If our API there were pure Go, it would require pulling in a huge amount of code from cmd/go/internal, and it would mean that we no longer use the Go version that the machine has available.
• If our API there used os/exec, it probably wouldn't use any other part of the x/mod module, and in my own mind would defeat the purpose of sharing the module

2. Long term, I'd like for this package to live somewhere like x/tools/modules. However, I don't want to rush that decision, especially given how difficult it was to stabilize x/tools/packages.

CC @jayconrod @matloob @ianthehat @myitcv @leitzler