I think we'd need a strong reason to consider adding more information. Also, many of the things you've asked for, like import path and documentation, are more about packages than modules. There's no guarantee that there is a package at the root of a module. Even licenses are something that pkg.go.dev shows per-package rather than per-module.

As for requires, downloading and parsing the go.mod shouldn't be that much of a burden? And that gives you the module name, which is essentially the import path you asked for.

@katiehockman @hyangah

I think we'd need a strong reason to consider adding more information. Also, many of the things you've asked for, like import path and documentation, are more about packages than modules. There's no guarantee that there is a package at the root of a module. Even licenses are something that pkg.go.dev shows per-package rather than per-module.

In my opinion, that maybe strong reason would be offering consistent metadata for modules and packages in universally readable format, allowing an easier interoperation with other package managers and analysis tools. Maybe not to the point of crates.io, but at least with enough information to build dependency graphs.

I've updated the proposal in order to remove references to packages and talk only about modules as, effectively, there were some misconceptions in the original issue.

As for requires, downloading and parsing the go.mod shouldn't be that much of a burden? And that gives you the module name, which is essentially the import path you asked for.

That's my last resort, though manually parsing sui generis formats from another programming language might not be the cleanest solution.

@heschik, I've solved my use case with a mix of regular expressions and proper selectors for extracting licenses and documentation from pkg.go.dev. 😅 Nevertheless, I've updated my proposal again to reflect a realistic package and module structure, including useful fields for external dependency managers.

For a bit of background, this proposal originated during a tentative refactoring of the Guix go-build-system. Though it's a niche use case, this proposal might have many other uses for tools that deal with Go packages and modules from another languages.

Feel free to close the issue if you deem it unproductive, though it would be no surprise finding a new use case for this functionality in a near future.

Thank you for your valuable advice!

The main purpose of proxy protocol is to serve the go command and we want to keep it minimal.

In my opinion, the proposed endpoint belongs to package/module discovery services such as pkg.go.dev.
#36785 is a relevant issue.
cc @julieqiu

💯 @hyangah, certainly, there is some overlap between the official module proxy and the official package/module discovery service, and #36785 is really apropos. Thank you!

If pkg.go.dev implements the proposed enhancement along with a dedicated code endpoint like .zip on the module proxy, the overlap would be complete, but we would have a solid registry/discovery service with the ability to serve a copy of both code and metadata for every known package in an easily readable format.

Do you think that proxying/caching useful metadata along with the code would be a good idea?

CC @bcmills, @jayconrod, @matloob per owners.

The main reason .info exists is so the go command has a way to resolve revision names like devbranch and 01234abcde to actual versions. When you run a command like go get example.com/mod@devbranch, it hits the .info endpoint to get back a canonical version, which it can then use with .mod and .zip.

One reason we're hesitant to extend the .info endpoint is that its content is not authenticated by go.sum or the checksum database. Proxies aren't trusted, and they're allowed to change what they serve for .info. So we especially don't want to serve anything redundant with .mod or .zip since that would entice users to discard a useful security guarantee.

About the specific fields you asked for:

• Documentation - there's no module-wide standard for documentation. pkg.go.dev shows README files from module zips. I think a few different extensions are recognized.
• Licenses - the module zip may include a file named LICENSE, though I think pkg.go.dev recognizes files with a few other extensions (.txt, and so on). We especially don't want to add anything in the proxy protocol automatically; that may put us and other implementers of the proxy protocol in the position of providing legal advice.
• Dependencies - already in go.mod. Use golang.org/x/mod/modfile to parse or use the grammar from the reference documentation.
• Hashes - use sum.golang.org instead. See Checksum database for protocol details.
• Additional fields - parse go.mod instead.

The main reason .info exists is so the go command has a way to resolve revision names like devbranch and 01234abcde to actual versions. When you run a command like go get example.com/mod@devbranch, it hits the .info endpoint to get back a canonical version, which it can then use with .mod and .zip.

That's a nice feature. In fact, I'm using it in order to resolve commit hashes and tags and get long versions for some modules.

One reason we're hesitant to extend the .info endpoint is that its content is not authenticated by go.sum or the checksum database. Proxies aren't trusted, and they're allowed to change what they serve for .info. So we especially don't want to serve anything redundant with .mod or .zip since that would entice users to discard a useful security guarantee.

That's a very interesting point: we can't obviate the importance of having integrity information for every module and its metadata, though relying on custom formats like go.sum and go.mod will force non-Go programs to parse them without the official parser in most situations.

Though it would be feasible to verify every package on the server and sign the JSON responses with JWS or similar methods, that would add server-side overhead and require a trusted proxy, directory or discovery service.

The only long-term solution I see for this issue would probably involve replacing both go.mod and go.sum with a JSON-based format, and this doesn't look like a good idea, given the insane amounts of packages that would need to change and the decrease in human readability of the new format when compared to the current one.

• Documentation - there's no module-wide standard for documentation. pkg.go.dev shows README files from module zips. I think a few different extensions are recognized.

I'm getting it from pkg.go.dev, but web scraping might not be the most elegant solution.

• Licenses - the module zip may include a file named LICENSE, though I think pkg.go.dev recognizes files with a few other extensions (.txt, and so on). We especially don't want to add anything in the proxy protocol automatically; that may put us and other implementers of the proxy protocol in the position of providing legal advice.

That's a really delicate issue, and the less friction path would probably be serving this information through pkg.go.dev by exposing a public API, as that site is already serving extracted license names in a nearly-SPDX format.

• Dependencies - already in go.mod. Use golang.org/x/mod/modfile to parse or use the grammar from the reference documentation.
• Additional fields - parse go.mod instead.

The issue is that golang.org/x/mod/modfile doesn't play nicely outside of the Go ecosystem, and reimplementations of custom formats might not be a good idea in this case.

• Hashes - use sum.golang.org instead. See Checksum database for protocol details.

I'll use it for now, thanks!

Closing this issue in favor of #36785, which seems like the best way to move forward in the long term.

I don't think implementing go.mod and go.sum parsers in other languages should be a huge barrier though. Both formats were chosen to be easily readable and editable by both humans and programs.

Completely agree, @jayconrod. Thanks!