Skip to content

cmd/go/internal/modfetch: re-evaluate the undocumented proxy behavior in proxyRepo.latest() #32789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
hyangah opened this issue Jun 26, 2019 · 4 comments
Open

cmd/go/internal/modfetch: re-evaluate the undocumented proxy behavior in proxyRepo.latest() #32789

hyangah opened this issue Jun 26, 2019 · 4 comments
Assignees
@matloob
Labels
Documentation Issues describing a change to documentation. GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@hyangah
Copy link
Contributor

hyangah commented Jun 26, 2019 •
edited
Loading

Forked from the discussion during the review of https://golang.org/cl/183402

  1. undocumented /@v/list format

proxyRepo.latest() assumes /@v/list could return a list of version, commit timestamp pairs. This is not documented in the proxy protocol doc, and I am not sure if there is any proxy that's following this format. Please reject the undocumented format or document it.

One of the benefit of preserving this feature is, when this proxyRepo.latest() is invoked, the commit timestamp info here is more reliable than the timestamp parsed from the pseudo-version like version string and helps avoiding an extra proxyRepo.Stat() call. On the other hand, the code path can be completely eliminated if proxies implement the /@latest endpoint, which is also not documented.

  1. whether to include pseudo-versions or not in /@v/list

The proxyRepo.latest() itself is also relying on part of undocumented behavior of /@v/list, which is supposed to include pseudo-versions. Before https://golang.org/cl/183402 (pre-1.13), we assumed it's not a good idea to include the pseudo versions in the /@v/list output and planned to include that in the proxy protocol spec.

https://go-review.googlesource.com/c/mod/+/176540/5/proxy/server.go#40

// List returns a list of tagged versions of the module identified by path.
// The versions should all be canonical semantic versions
// and formatted in a text listing, one per line.
// Pseudo-versions derived from untagged commits should be omitted.
// The go command exposes this list in 'go list -m -versions' output
// and also uses it to resolve wildcards like 'go get m@v1.2'.`

If the spec draft is valid, the proxyRepo.latest() implementation doesn't do anything but an extra network round trip for the proxies that follow the spec. If the go command can handle the list output including pseudo-versions, we may want to revisit the decision.

So, I propose either

  1. tighten the spec to require the /@latest endpoint. Then, remove the code that leads to the proxyRepo.latest(), or
  2. relax the /@v/list format and allow pseudo versions in its output, and document it.

Option 2) implies more code to maintain on cmd/go side. Option 1) implies breakage of proxies that doesn't implement the /@latest endpoint.

p.s. having the go command's proxyRepo.Versions ensure not to return pseudo versions is a good fix though.
@gopherbot gopherbot added the Documentation Issues describing a change to documentation. label Jun 26, 2019
@bcmills

This comment has been minimized.

Sign in to view
@bcmills bcmills added this to the Go1.14 milestone Jun 26, 2019
@andybons andybons added the NeedsFix The path to resolution is known, but the work has not been done. label Jun 28, 2019
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 12, 2019
@gopherbot gopherbot removed the NeedsFix The path to resolution is known, but the work has not been done. label Aug 12, 2019
@dmitshur
Copy link
Contributor

dmitshur commented Aug 16, 2019 •
edited
Loading

  1. whether to include pseudo-versions or not in /@v/list

As an observation, the last paragraph at https://golang.org/cmd/go/#hdr-Module_proxy_protocol says:

Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy. The cache layout is the same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) https://example.com/proxy would let other users access those cached module versions with GOPROXY=https://example.com/proxy.

If one looks inside the local module cache as produced by cmd/go, pseudo-versions are included in /@v/list output. E.g.:

$ cat $(go env GOPATH)/pkg/mod/cache/download/golang.org/x/text/@v/list
v0.3.0
v0.3.1-0.20180807135948-17ff2d5776d2
v0.3.1-0.20181227161524-e6919f6577db
v0.3.2

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/190838 mentions this issue: cmd/go: document <module>/@latest endpoint in 'go help goproxy'

@jayconrod
Copy link
Contributor

  1. undocumented /@v/list format

I think we should confirm no proxies are using extra fields. If there are none, we should start rejecting them in 1.14.

Currently, it looks like this can be used to set timestamps. We only really need timestamps for ordering pseudo-versions, and the timestamp should match the timestamp encoded in the pseudo-version, so there's no need to specify it separately.

  1. whether to include pseudo-versions or not in /@v/list

CL 190838 updates go help goproxy to include /@latest. It also points out that /@v/list may include pseudo-versions.

Note that go1.13 treats /@v/list differently from go1.12 and earlier.

In go1.13, if /@v/list contains tagged versions and pseudo-versions, the pseudo-versions will only be considered if all tagged versions (including pre-release) are excluded.

In go1.12 and earlier, if /@v/list contains tagged versions and pseudo-versions, the pseudo-versions will be considered together with tagged pre-release versions (non-pre-release versions still take precedence).

So if a proxy stopped serving /@latest and instead included that version in /@v/list (if not already present), it would change behavior of go1.12 and earlier if that version is a pseudo-version newer than pre-release versions in /@v/list. I don't think that change would affect proxy.golang.org since a pseudo-version is only returned from /@latest if /@v/list is empty.

gopherbot pushed a commit that referenced this issue Sep 12, 2019
cmd/go: document <module>/@latest endpoint in 'go help goproxy'

Verified

This commit was signed with the committer’s verified signature.
renovate-bot Mend Renovate
GPG key ID: A1020E7F5FFD673D
Verified
Learn about vigilant mode
9428861 
Updates #32789

Change-Id: Ie5e8e3b7b6a923aa9068c8af3ac8f081bd92c830
Reviewed-on: https://go-review.googlesource.com/c/go/+/190838
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@seankhliao seankhliao added the GoCommand cmd/go label Feb 22, 2025
@matloob matloob self-assigned this Apr 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@matloob matloob

Labels
Documentation Issues describing a change to documentation. GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
9 participants
@rsc @jayconrod @andybons @dmitshur @hyangah @bcmills @gopherbot @seankhliao @matloob