This seems like an implementation detail of the module server. The protocol for a server and a proxy is the same, but a server that is not a proxy fundamentally needs to know which paths it's serving.

(Note that one option is for the server to run go mod download, but on path that differs from the one requested by the user.)

For the @latest path in particular, I suspect that @hyangah, @heschik, and/or @katiehockman can give you more useful advice.

This is indeed an implementation detail of the module server.
This issue brings up an interesting attack scenario against certain public proxies. That's also proxy implementation details, and public proxies should be prepared to handle this type of unexpected use or misuse cases.

The /@latest endpoint is not crucial for the go command to operate as long as /@v/list returns a non-empty list. If the storage server is implemented as described in the original vgo proposal, not having /@latest endpoint will not be an issue.

@hyangah In this case it should be 100% safe to say that the only way a proxy server knows how to go mod download a vanity import path with a mod meta tag, is by knowing its counter-part VCS import path ahead of time.

For example, if a user did:

go get marwan.io/moddoc

And then my marwan.io server returned something like ~<meta mod https://someproxy.com>

Then my myproxy.com must have an ahead of time knowledge that marwan.io maps to github.com/marwan-at-work so that it will do the correct go mod download without getting stuck in an infinite loop. Assuming go mod download github.com/marwan-at-work/moddoc should never fail because the source code's import paths start with marwan.io/moddoc

The only other way around my solution above, is that the proxy server has a way for that vanity import path to be populated outside of go mod download, i.e. by manually uploading its own .zip/.info/.mod without using go mod download which would run the risk of checksum mismatches and that proxy has to be the only place to ever get that module.

If all of my assumptions are correct, I'm happy closing the issue :)

Thanks!

@marwan-at-work a different approach to protect the public proxy from the infinite loop is to design the myproxy.com to avoid duplicate work. The subsequent request in the chain will then be classified as duplicate of pending operation, and not add additional work to the proxy but wait for the pending duplicate work to complete. Eventually some of the request in the chain will timeout (as most production service would do) and the chain will terminate.

@hyangah I'm more focused on how it should work as opposed to the security vulnerability side of it.

From a vulnerability stand point, your suggestion makes sense. Another one is that the proxy server can also issue its own GET marwan.io/moddoc?go-get=1 and check that its mod <meta> tag is not pointing to the same host as the proxy itself.

However, from a feature perspective, I wasn't clear on how go mod download and the mod <meta> tag should work with each other besides your suggestion of go mod download VCS-PATH.

But I'd love a confirmation that the mod <meta> tag feature is indeed intended for my two suggestions in my previous comment (go mod download vcs-path (not vanity path) and direct upload) by design.

Thanks again :)

@marwan-at-work One of the use cases where the mod go-import meta tag feature seems to work well is for publishing modules that do not have a counter-part public VCS.

For example, the module dmitri.shuralyov.com/test/a has a vanity import path and it uses the mod type, but there is no public accompanying VCS. New versions of that module are published directly to the module proxy that is providing it. The module proxy is just a static filesystem being served over HTTP, using http.FileServer. As a result, this problem does not come up in that setup.

I agree that it becomes problematic if the mod type is meant to be used to augment a module with a vanity import path that has an underlying public VCS and new versions of the module are created by pushing to the VCS. I'm not currently sure how that should be handled.

Note that go mod download vcs-path isn't viable, either. The resultant ZIP archive will contain file paths prefixed by the VCS path.

looks like the answer is clear: the proxy implementation should not invoke go for which it is the authoritative source.