x/pkgsite: support retractions #43265

jba · 2020-12-18T12:11:21Z

Beginning with Go 1.16, module authors will be able to retract versions of their modules. The go command will ignore retracted versions by default.

pkg.go.dev should expose that a version is retracted.

UI Ideas

The page for a retracted version should show that it is retracted, and display the rationale if any.
In the list of module versions, retracted versions should either be displayed at the bottom, or indicated in some other way (such as a red strikethrough). They shouldn't be hidden, because someone might be using a retracted version and want to visit its page.
We could show which imported packages are retracted, but that depends on having precise version information about imports (that is, the build list), which we don't currently have.

Implementation

The proposal for retraction is at #24031 (comment). In brief: retraction is done by adding retract directives in go.mod files. Each directive lists a range of versions and a rationale. Only the latest go.mod file, as defined by the go command's @latest query, is authoritative.

The modules table should have a column that contains the information in the retract directives, perhaps as JSON. The worker parses the go.mod file and populates the column during processing.
The frontend reads the retractions from the latest version of a module and applies them to the module in the request.

This involves an extra DB query sometimes, but only when viewing a version that is not the latest, which is uncommon.

One wrinkle: the latest version can retract itself. In that case, finding the true latest version (the one that deserves the "LATEST" badge) requires another pass over the list of unretracted versions.

In an alternative design, there is a column modules.retracted which gives the state for each version, and the worker populates that for all versions when it is processing the latest one. This involves some subtleties:

Writing rows other than the current one can result in deadlocks.
The worker can process versions in any order, so it might see a version in a retract directive before it has processed it. Where should it store that information?

The advantages are fewer queries at serving time in some cases, and the ability to query easily for the latest unretracted version.

The text was updated successfully, but these errors were encountered:

gopherbot · 2021-02-23T12:33:07Z

Change https://golang.org/cl/295430 mentions this issue: internal/proxydatasource: implement deprecation and retractions

When fetching a module at a version, the proxy datasource uses fetch.RawLatestInfo to get the go.mod file at the raw latest version of the module, then uses internal.RawLatestInfo.PopulateModule to determine whether the module version is deprecated or retracted. Also, add some proxy test modules to facilitate testing. For golang/go#41321 For golang/go#43265 For golang/go#44437 Change-Id: I312346d72f656e598ad170135046ef85da8e9b11 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295430 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-02-24T14:20:27Z

Change https://golang.org/cl/295891 mentions this issue: internal: add experiment for retractions and deprecations

gopherbot · 2021-02-24T14:52:47Z

Change https://golang.org/cl/295892 mentions this issue: internal/postgres: add retraction/deprecation info to GetModuleInfo

For golang/go#43265 For golang/go#41321 Change-Id: I1e8e6fe0f55d536696e1c58e81726292278fd20c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295891 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

For golang/go#43265 For golang/go#41321 Change-Id: I5812501765ffd2a2ce57df6795ccd558884d3087 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295892 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-02-24T19:03:02Z

Change https://golang.org/cl/295896 mentions this issue: internal/postgres: add retraction/deprecation info to versions

gopherbot · 2021-02-24T19:52:18Z

Change https://golang.org/cl/295898 mentions this issue: internal: compute deprecation when RawLatestInfo is created

gopherbot · 2021-02-24T19:52:18Z

Change https://golang.org/cl/295897 mentions this issue: internal: add NewRawLatestInfo method

gopherbot · 2021-02-24T19:58:52Z

Change https://golang.org/cl/295899 mentions this issue: internal/postgres: add retraction/deprecation info to GetNestedModules

For golang/go#43265 For golang/go#41321 Change-Id: I54f81bcc6b2e4cf31aacdbd630cc0650a833ae25 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295896 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

Consolidate creation of RawLatestInfo in one function. For golang/go#43265 Change-Id: Id57d044c33a678b8ff4e57ada1a5569e09f841d0 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295897 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

The deprecation information depends only on the go.mod file, so we can compute it once in NewRawLatestInfo. For golang/go#43265 Change-Id: I1626ab39ff776450147c3e4a4b33883ab2910b2c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295898 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

For golang/go#43265 Change-Id: I297856d635eda30d09e42a860f39e0cffc452464 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295899 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-02-25T00:06:54Z

Change https://golang.org/cl/296209 mentions this issue: internal: embed ModuleInfo into UnitMeta

gopherbot · 2021-02-25T00:34:57Z

Change https://golang.org/cl/296229 mentions this issue: internal/postgres: add retraction/deprecation info to GetUnitMeta

UnitMeta had most of the fields of ModuleInfo, but did not embed it. Now that we have added four more fields to ModuleInfo for deprecation and retraction, it makes sense to embed it rather than duplicate those fields. For golang/go#43265 For golang/go#41321 Change-Id: I20e2b922b49c7873a5535745d644631123de37cd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296209 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

For golang/go#43265 For golang/go#41321 Change-Id: I8dc86d87882b96d3c3ffe7c1d035937f210b9a91 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296229 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-02-26T14:49:27Z

Change https://golang.org/cl/296812 mentions this issue: internal/postgres: truncate raw_latest_versions

gopherbot · 2021-02-26T14:49:29Z

Change https://golang.org/cl/296815 mentions this issue: content/static: add deprecated and retracted banners

For golang/go#43265 For golang/go#41321 Change-Id: I7bf803b68b1532b968ad1175433275f5a4778078 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296812 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

When a module is deprecated or retracted, add a banner to the main page. Test with an integration test to verify that the deprecation/retraction information makes it through the entire pipeline. For golang/go#43265 For golang/go#41321 Change-Id: I0ffd1a9d1617e2865a10f0b0a8a8a3af6ed4d420 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296815 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-03-01T13:14:38Z

Change https://golang.org/cl/297509 mentions this issue: internal/frontend: add deprecated/retracted info to versions

For golang/go#43265 Change-Id: Iac39814ce532adf5846bb768802a46ad7a77fa84 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/309609 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-04-13T11:26:50Z

Change https://golang.org/cl/309610 mentions this issue: internal/postgres: remove uses of modules.deprecated_comment

gopherbot · 2021-04-13T13:38:15Z

Change https://golang.org/cl/309709 mentions this issue: internal: remove unit-meta-with-latest experiment

gopherbot · 2021-04-13T13:45:44Z

Change https://golang.org/cl/309710 mentions this issue: migrations: drop modules.deprecated_comment

The deprecated_column in the modules table has been superseded by the information in the latest_module_versions table. For golang/go#43265 Change-Id: Ib53e0b295a3edf8e807ff825b36baa6701b927b1 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/309610 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

For golang/go#43265 Change-Id: I1d056a893fdff60744ff328ab9f4a1b6665b3e32 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/309709 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

For golang/go#43265 Change-Id: I5885853463ed52915a7956e66bdfae9498ce90df Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/309710 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

jba · 2021-04-19T16:19:55Z

Retraction support is live.

zx2c4 · 2021-04-19T17:52:19Z

Apparently it doesn't see the retractions in https://pkg.go.dev/golang.zx2c4.com/wireguard . The v0.0.20201119 version should retract all others and itself: https://git.zx2c4.com/wireguard-go/tree/go.mod?h=v0.0.20201119

jba · 2021-04-21T13:26:49Z

There's something odd going on. Did you perhaps change a tag?

According to the Go tools, the latest version of your module is indeed v0.0.20201119. But the proxy won't download that version, and here's what happens when I try to get it directly:

> GOPROXY=direct go get golang.zx2c4.com/wireguard@v0.0.20201119
go: downloading golang.zx2c4.com/wireguard v0.0.20201119
go get: golang.zx2c4.com/wireguard@v0.0.20201119: verifying module: checksum mismatch
        downloaded: h1:BbmPvlD3yvKCF1ybLUCHD+JYFn3my0MKPY24rJi41bY=
        sum.golang.org: h1:WJ4IKfT3SLG+YbM9aeZlgYB+X7hKzO66GEGBmxJPhjE=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.

The best approach would be to tag a new, later version that has a go.mod with all the desired retractions.

zx2c4 · 2021-04-23T02:35:29Z

You're right. I had changed a tag, after I realized that proxy.golang.org won't cache things without at least a single .go file beyond a few days. So that explains that.

Current status is that retractions don't seem to be propagating:

All of those should actually be retracted, per the go.mod file:

zx2c4 · 2021-05-05T12:16:30Z

It looks like it's still very goofed up:

It thinks that the most recent commit is https://git.zx2c4.com/wireguard-go/commit/?id=6a128dd which is some random old commit from January, rather than the most recent commit, which is today.

jba · 2021-05-06T14:08:35Z

Sorry for the delay getting back to you.

Your module is an interesting case. You've retracted all the tagged versions, leaving only pseudo-versions. And some of those pseudo-versions, having been landed on top of tagged versions, are technically "later" than the more recent ones you've published.

Can I suggest tagging the version you want to be latest with a new, higher tag? Tagged versions are definitely the way to go.

zx2c4 · 2021-05-06T14:15:54Z

You've retracted all the tagged versions, leaving only pseudo-versions.

I'm following precisely the suggestion given by the Go team earlier for retracting all versions. Things work perfectly with go get. CC @bcmills

Can I suggest tagging the version you want to be latest with a new, higher tag? Tagged versions are definitely the way to go.

The project is not yet ready to release a stable version. Therefore no tags are possible at the moment. Rather, the logic in pkgsite is clearly borked, as go get does the right thing.

jba · 2021-05-06T14:34:28Z

Agreed, pkgsite has a bug. We'll work on fixing it.

However, tags don't imply stability. You can tag with a major version of zero.

jba · 2021-05-06T15:04:28Z

Actually, I'm not sure that the bug is in pkgsite. Or at least, I don't quite know what the bug is.

To go over the logic again:

Since all the tagged versions are retracted, only the pseudo-versions matter.
Pseudo-versions are ordered by semver.
The most recent pseudo-version begins v0.0.0, but an older one has a higher version. The older is in the proxy's index and can be downloaded from the proxy:

> go get golang.zx2c4.com/wireguard@v0.0.20201119-0.20210128142622-6a128dde71d9
go: downloading golang.zx2c4.com/wireguard v0.0.20201119-0.20210128142622-6a128dde71d9
go get: upgraded golang.zx2c4.com/wireguard v0.0.0-20210506092213-60a26371f42f => v0.0.20201119-0.20210128142622-6a128dde71d9

Note that the go command thinks it is "upgrading" the version.

The difference between the go command and pkgsite is that the former knows only what the proxy tells it, and the proxy believes that the v0.0.0 pseudo-version is the latest. I'm not sure why. @heschi @hyangah, is there logic to ignore pseudo-versions derived from retracted tagged versions? More likely, the proxy @latest endpoint is just tracking the main/master branch.

Pkgsite has the full history so it "correctly" serves the highest version.

@jayconrod @julieqiu What's the right fix here? Maybe pkgsite should ignore pseudo-versions derived from retracted versions?

heschi · 2021-05-06T15:08:15Z

We just report whatever go get @latest says. Based on what you're saying, I presume that retracted versions no longer serve as pseudoversion bases, so after the 20201119 tag was retracted the pseudoversions "downgraded".

As I think we've discussed elsewhere, AFAIK the only way to know for sure what the "latest" version is is to ask the go command each time.

bcmills · 2021-05-06T16:35:18Z

Based on what you're saying, I presume that retracted versions no longer serve as pseudoversion bases, so after the 20201119 tag was retracted the pseudoversions "downgraded".

That is correct. Ideally, when you retract an erroneous release version you should also retract the range of pseudo-versions derived from that release, although in some sense those pseudo-versions were never “released” to begin with.

gopherbot · 2021-05-07T18:38:16Z

Change https://golang.org/cl/318069 mentions this issue: internal/postgres: ensure good version not later than cooked

The good latest version of a module should never be later than the cooked latest version. If it is, then pkgsite will show a different version than the go command will download. One way this can happen is if a module retracts all tagged versions, and pseudo-versions were built on top of some of them. For example, initially it could have versions v0.1.0 v0.1.0-DATE-HASH (a pseudo-version) Then v0.1.0 is retracted and a new pseudo-version appears at head: v0.1.0 (retracted) v0.1.0-DATE-HASH v0.0.0-DATE-HASH (head of default branch) The go command will get the v0.0.0 version, but technically the v0.1.0 is later. For golang/go#43265 Change-Id: I8ff30de4eb2dcdf108205de99af93d2f31772cff Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318069 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

jba · 2021-05-18T13:45:44Z

This should be fixed. https://pkg.go.dev/golang.zx2c4.com/wireguard shows a version published on May 13, a few days ago. Reopen if there are still problems.

zx2c4 · 2021-05-18T13:46:39Z

Excellent, thank you for fixing this!

jba added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. FeatureRequest pkgsite labels Dec 18, 2020

jba added this to the pkgsite/unplanned milestone Dec 18, 2020

jba self-assigned this Dec 18, 2020

jba mentioned this issue Feb 19, 2021

x/pkgsite: handle raw latest versions #44437

Closed

seankhliao mentioned this issue Feb 28, 2021

x/pkgsite: mismatched version, tags, LATEST #44694

Closed

jba closed this as completed Apr 19, 2021

jba reopened this Apr 21, 2021

seankhliao mentioned this issue May 4, 2021

x/mod: Add a keyword to go.mod to delete or ignore in pkg.go.dev #45936

Closed

jba closed this as completed May 18, 2021

liggitt mentioned this issue Nov 30, 2021

k8s.io: Redirect common Go package URLs to pkg.go.dev kubernetes/k8s.io#3130

Closed

golang locked and limited conversation to collaborators May 18, 2022

gopherbot added the FrozenDueToAge label May 18, 2022

rsc unassigned jba Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/pkgsite: support retractions #43265

x/pkgsite: support retractions #43265

jba commented Dec 18, 2020

gopherbot commented Feb 23, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 25, 2021

gopherbot commented Feb 25, 2021

gopherbot commented Feb 26, 2021

gopherbot commented Feb 26, 2021

gopherbot commented Mar 1, 2021

gopherbot commented Apr 13, 2021

gopherbot commented Apr 13, 2021

gopherbot commented Apr 13, 2021

jba commented Apr 19, 2021

zx2c4 commented Apr 19, 2021

jba commented Apr 21, 2021

zx2c4 commented Apr 23, 2021

zx2c4 commented May 5, 2021

jba commented May 6, 2021

zx2c4 commented May 6, 2021

jba commented May 6, 2021

jba commented May 6, 2021

heschi commented May 6, 2021

bcmills commented May 6, 2021

gopherbot commented May 7, 2021

jba commented May 18, 2021

zx2c4 commented May 18, 2021

x/pkgsite: support retractions #43265

x/pkgsite: support retractions #43265

Comments

jba commented Dec 18, 2020

UI Ideas

Implementation

gopherbot commented Feb 23, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 24, 2021

gopherbot commented Feb 25, 2021

gopherbot commented Feb 25, 2021

gopherbot commented Feb 26, 2021

gopherbot commented Feb 26, 2021

gopherbot commented Mar 1, 2021

gopherbot commented Apr 13, 2021

gopherbot commented Apr 13, 2021

gopherbot commented Apr 13, 2021

jba commented Apr 19, 2021

zx2c4 commented Apr 19, 2021

jba commented Apr 21, 2021

zx2c4 commented Apr 23, 2021

zx2c4 commented May 5, 2021

jba commented May 6, 2021

zx2c4 commented May 6, 2021

jba commented May 6, 2021

jba commented May 6, 2021

heschi commented May 6, 2021

bcmills commented May 6, 2021

gopherbot commented May 7, 2021

jba commented May 18, 2021

zx2c4 commented May 18, 2021