x/pkgsite: module latest-version information #44710

jba · 2021-03-01T13:55:17Z

Pkgsite needs to track three kinds of latest versions for each module:

The raw latest version (see x/pkgsite: handle raw latest versions #44437)—determines retractions and deprecations;
The cooked latest version, which is the raw one after applying retractions—what go get will download;
The good latest version, which is the latest version that pkg.go.dev has useful information for.

We will have a single table, module_latest_versions, to track all of these. The table will be updated by the worker each time a module (at any version) is processed. It will be consulted by GetUnitMeta and other DataSource methods that require the latest version of a module.

Usually all three kinds of latest version will be the same, but sometimes they will differ. Pkgsite will always show the good latest version when the URL is unversioned (that is, it does not contain "@version"). If the cooked latest version differs, the page will say so.

The text was updated successfully, but these errors were encountered:

gopherbot · 2021-03-01T21:15:56Z

Change https://golang.org/cl/297649 mentions this issue: internal/version: generalize Later to pseudo-versions

gopherbot · 2021-03-01T21:25:53Z

Change https://golang.org/cl/297689 mentions this issue: migrations: add latest_module_versions table

The Later function can now handle pseudo-versions. For golang/go#44710 Change-Id: I19249c92f923ec759c859905ef547c2460760950 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/297649 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>

For golang/go#44710 Change-Id: Ifc590cdb4abd68a3d0210c8d5f9248154fbfd19a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/297689 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>

gopherbot · 2021-03-02T13:58:38Z

Change https://golang.org/cl/297849 mentions this issue: internal: add LatestModuleVersions

gopherbot · 2021-03-02T16:08:40Z

Change https://golang.org/cl/297889 mentions this issue: internal/version: implement full latest logic

The new LatestModuleVersions struct generalizes RawLatestInfo with information about the cooked and good latest versions. It will eventually replace RawLatestInfo. For golang/go#44710 Change-Id: I2c5db352e3ff27b1fd003c677c4b2273db111425 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/297849 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-03-03T13:17:20Z

Change https://golang.org/cl/298229 mentions this issue: internal/postgres: get/update LatestModuleVersions

gopherbot · 2021-03-03T14:17:31Z

Change https://golang.org/cl/298230 mentions this issue: internal/fetch: compute latest module versions from proxy

gopherbot · 2021-03-03T14:47:35Z

Change https://golang.org/cl/298231 mentions this issue: internal/worker: update latest versions on each fetch

gopherbot · 2021-03-03T15:02:40Z

Change https://golang.org/cl/298232 mentions this issue: internal/postgres: use latest versions to populate ModuleInfo

gopherbot · 2021-03-03T15:07:57Z

Change https://golang.org/cl/298233 mentions this issue: internal/proxydatasource: use LatestModuleVersions

gopherbot · 2021-03-03T16:17:49Z

Change https://golang.org/cl/298234 mentions this issue: many: remove raw-latest code

@latest

Implement the full latest-version logic in internal/versions, because we'll need it in a couple of places. Also generalize it to a single list of versions that may include pseudo-versions. That way it's not tied to the proxy's two endpoints (list and @latest) and can be used on any list of versions, like a list of all module versions. For golang/go#44710 Change-Id: I82ad9ae1e585762d13fecb1997c89312b88ad54d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/297889 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

Add methods to read and update the latest_module_versions table using internal.LatestModuleVersions. For golang/go#44710 Change-Id: I5e360933aeb8b8ea1c92204872d093c370de8eaf Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298229 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

For golang/go#44710 Change-Id: I04274e5f06f1c9fcd6029c27bf692821a7e0e50a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298230 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

Update latest-version information in the DB after we successfully process a module, even if the module is not inserted. For golang/go#44710 Change-Id: Ib7dfc64187882f96b870622df5faebd69d2ef2bf Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298231 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

Replace the use of the raw_latest_versions table with the latest_module_versions table. For golang/go#44710 Change-Id: I69eff3f19d209e45d5a568209087eca9cf13a0f2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298232 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

For golang/go#44710 Change-Id: I070c7f446e94511006d1d97b0bdd6a380e012c63 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298233 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

The raw-latest system is replaced by the latest-version system. For golang/go#44710 Change-Id: Id59f03aa3078c05e2a91b58d92633cc73dea4c3e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298234 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-03-04T00:35:38Z

Change https://golang.org/cl/298510 mentions this issue: internal/postgres: move GetUnitMeta to legacyGetUnitMeta

gopherbot · 2021-03-04T00:52:14Z

Change https://golang.org/cl/298511 mentions this issue: internal/postgres: use latest good module version in GetUnitMeta

gopherbot · 2021-03-04T12:23:55Z

Change https://golang.org/cl/298629 mentions this issue: internal/fetch: handle lack of version information

gopherbot · 2021-03-04T16:20:22Z

Change https://golang.org/cl/298729 mentions this issue: internal: add LatestModuleVersions.IsRetracted method

The orderByLatest function doesn't always produce the right latest version, so we will remove it, bit by bit. This CL rewrites the query in getLatestMajorVersion to avoid the orderByLatest function. For this particular case, a simpler order-by clause will work, and there is no need to deal with retractions. For golang/go#44710 Change-Id: I80da4f98871423447a9c5af2f8b9004a5d67a597 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/301210 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

…eVersion Remove orderByLatest from another function. It does not produce accurate results. In the case of getLatestMinorModuleVersion, we can use the latest good version of the module directly if it's available. Also, rename getLatestMinorModuleVersion to the more meaningful unitExistsAtLatest. For golang/go#44710 Change-Id: Ic438c68327570ed7dc4c17825ec48f1ad1109d46 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/301211 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-03-15T19:06:10Z

Change https://golang.org/cl/301929 mentions this issue: internal/worker,etc.: improvements to latest-version info

- Fetch and update the latest-version info before processing a module rather than after, so the information in the DB matches what InsertModule is using. - Pass latest-version info to InsertModule so it can use it to decide whether the version being inserted is the latest version. - InsertModule writes the good latest version to the latest_module_versions table while holding the advisory lock, avoiding a race condition where two different versions both think they're the latest. For golang/go#44710 Change-Id: Id5e8fa7a384ec6d4d86257362fd33a8fcd5215f7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/301929 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-03-16T21:23:42Z

Change https://golang.org/cl/302429 mentions this issue: internal/postgres: update latest module versions when cooked changes

In addition to the cases considered already, update the latest-version information for a module if the raw version hasn't changed but the cooked version has. For golang/go#44710 Change-Id: I88091433ca236bf9a5e03db2dd3bc751134f912d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/302429 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-03-17T12:38:33Z

Change https://golang.org/cl/302529 mentions this issue: internal/postgres: check for go.mod includes module_version_states

gopherbot · 2021-03-17T13:08:15Z

Change https://golang.org/cl/302530 mentions this issue: internal/fetch: get latest-version info for stdlib

gopherbot · 2021-03-17T13:11:35Z

Change https://golang.org/cl/302531 mentions this issue: migrations: drop the raw_latest_versions table

gopherbot · 2021-03-17T14:10:01Z

Change https://golang.org/cl/302532 mentions this issue: internal/proxy: support caching the last zip

jba · 2021-03-17T18:47:39Z

One important piece that is missing from this: if we ever learn that the good latest version changes due to a retraction, we need to re-do the imports_unique and search_documents tables for that module.

I'm not talking about the normal case, where a new, latest version is published and we update those two tables while we process it. This is a new case we haven't dealt with before: a new raw latest go.mod retracts the latest good version (and itself), leaving an older version as the latest good one.

Say the current latest (raw, cooked and good) is v1.2.0, and there is also a v1.1.0 in the DB. Now v1.3.0 is published, with a go.mod that retracts v1.3.0 and v1.2.0.

When we process v1.3.0, we need to notice that the latest good version has changed (to v1.1.0), and that it is not the version we're currently processing. We then need to update latest_module_versions.good_version, and repopulate the two tables mentioned above.

We sometimes need to know whether a a particular module version's zip has a go.mod file in order to determine the latest version of the module. As an optimization, we check the DB first before getting the zip from the proxy. Up until now, we checked only the modules table, which includes only "good" modules. This CL extends that check to the module_version_states table, which includes every version we've seen. For golang/go#44710 Change-Id: If3381615ac30c0dfc0706f9e4b14ed387fad25b2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/302529 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>

Determine and store the latest=version information of the standard library, instead of treating it as a special case everywhere else. For golang/go#44710 Change-Id: I09a20a12f9758b48a494f867bf1e1376bb042c0f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/302530 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

It is superseded by the latest_module_versions table. For golang/go#44710 Change-Id: Ia25c250ef0b04d064123848229f5070005915674 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/302531 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

To avoid downloading a zip twice when processing a module, implement a one-element zip cache in the proxy client. The test in internal/worker/fetch_test.go explains in more detail when this is useful. For golang/go#44710 Change-Id: I89e67a5452833a7300fc2e10981e2b59594f10a9 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/302532 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>

gopherbot · 2021-03-19T14:11:29Z

Change https://golang.org/cl/303309 mentions this issue: internal/postgres: reimplement alternative-module check

gopherbot · 2021-03-19T17:31:57Z

Change https://golang.org/cl/303311 mentions this issue: internal/postgres: update latest good when it is retracted

When inserting a module, we skip inserting into search_documents if the module path is an alternative one. This CL factors out that test, and also reimplements it in terms of the latest cooked version. Previously, any later version with a 491 status would classify the path as alternative, but this version uses the latest cooked (non-retracted) version's status to decide, allowing module authors to fix mistakes. Change-Id: I3588f3df6a1a4dd4db59333314c34167eda51df1 For golang/go#44710 Change-Id: I670d9085800fed198e6dd6469f1a8d5728c5f039 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303309 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>

gopherbot · 2021-03-19T18:35:32Z

Change https://golang.org/cl/303312 mentions this issue: internal/postgres: add ReInsertLatestVersion

Previously, UpdateLatestModuleVersions only changed the raw and cooked versions, leaving the good version to InsertModule. Now it updates the good version when the current good version is retracted, which otherwise would not happen. For golang/go#44710 Change-Id: I90d771310dbfb56ae02255e0abc3afddcce7b34c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303311 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>

gopherbot · 2021-03-19T20:59:10Z

Change https://golang.org/cl/303313 mentions this issue: internal/postgres: add imports_unique to ReInsertLatestVersion

New latest-version info can retract the current latest good version, resulting in an older version becoming the new latest good version. Add a function that updates the search_documents table when that happens. A later CL will also update imports_unique, and will call this function from the worker. For golang/go#44710 Change-Id: I09569e27d384ed7defb06a4e81facdb96c22caeb Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303312 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Julie Qiu <julie@golang.org>

The ReInsertLatestVersion method now changes imports_unique as well as search_documents. For golang/go#44710 Change-Id: Ie5dcfe8e362313be76f0ad4958882e37fc725e44 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303313 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>

gopherbot · 2021-03-22T17:12:57Z

Change https://golang.org/cl/303649 mentions this issue: internal/worker: call ReInsertLatestVersion

After processing a module, call ReInsertLatestVersion. Skip the call if this is the cooked latest version of an alternative path. The same check is made inside ReInsertLatestVersion, but using the module_version_states table, and the module being processed has not yet been inserted into that table. For golang/go#44710 Change-Id: Icd0d0c0045ccadf3e97d32c63146262a3b442577 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303649 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>

jba · 2021-04-05T19:18:42Z

We are now tracking and using latest-version information.

gopherbot added the pkgsite label Mar 1, 2021

jba self-assigned this Mar 1, 2021

gopherbot added this to the Unreleased milestone Mar 1, 2021

jba modified the milestones: Unreleased, pkgsite/unplanned Mar 1, 2021

julieqiu added the NeedsFix The path to resolution is known, but the work has not been done. label Mar 2, 2021

jba closed this as completed Apr 5, 2021

golang locked and limited conversation to collaborators Apr 5, 2022

gopherbot added the FrozenDueToAge label Apr 5, 2022

rsc unassigned jba Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/pkgsite: module latest-version information #44710

x/pkgsite: module latest-version information #44710

jba commented Mar 1, 2021

gopherbot commented Mar 1, 2021

gopherbot commented Mar 1, 2021

gopherbot commented Mar 2, 2021

gopherbot commented Mar 2, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 4, 2021

gopherbot commented Mar 4, 2021

gopherbot commented Mar 4, 2021

gopherbot commented Mar 4, 2021

gopherbot commented Mar 15, 2021

gopherbot commented Mar 16, 2021

gopherbot commented Mar 17, 2021

gopherbot commented Mar 17, 2021

gopherbot commented Mar 17, 2021

gopherbot commented Mar 17, 2021

jba commented Mar 17, 2021 •

edited

gopherbot commented Mar 19, 2021

gopherbot commented Mar 19, 2021

gopherbot commented Mar 19, 2021

gopherbot commented Mar 19, 2021

gopherbot commented Mar 22, 2021

jba commented Apr 5, 2021

x/pkgsite: module latest-version information #44710

x/pkgsite: module latest-version information #44710

Comments

jba commented Mar 1, 2021

gopherbot commented Mar 1, 2021

gopherbot commented Mar 1, 2021

gopherbot commented Mar 2, 2021

gopherbot commented Mar 2, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 3, 2021

gopherbot commented Mar 4, 2021

gopherbot commented Mar 4, 2021

gopherbot commented Mar 4, 2021

gopherbot commented Mar 4, 2021

gopherbot commented Mar 15, 2021

gopherbot commented Mar 16, 2021

gopherbot commented Mar 17, 2021

gopherbot commented Mar 17, 2021

gopherbot commented Mar 17, 2021

gopherbot commented Mar 17, 2021

jba commented Mar 17, 2021 • edited

gopherbot commented Mar 19, 2021

gopherbot commented Mar 19, 2021

gopherbot commented Mar 19, 2021

gopherbot commented Mar 19, 2021

gopherbot commented Mar 22, 2021

jba commented Apr 5, 2021

jba commented Mar 17, 2021 •

edited