Define helper functions that implement a part of the go
command's definition of "latest": preferring release
to pre-release versions.

We will use Latest in subsequent CLs to implement
the full definition.

For golang/go#44437

Change-Id: I56f6448292d1ff373266da03d830e2882e6016c3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295193
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

Add fields to hold retraction information.

Also, switch deprecation to a pair of bool and string rather than a
pointer, because it's clearer.

For golang/go#44437

Change-Id: Ica771c5ec8e5c0abeec16f9d998339ef60b401a5
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295195
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

Add RawLatestInfo, which describes the raw latest version of a module,
and provide functions to compute deprecation and retractions from it.

For golang/go#44437

Change-Id: I2f1aa03bf190b961642828fdc7b708e2ee47a20a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295196
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

Add RawLatestInfo, which uses the proxy to find the raw latest version
of a module and its associated go.mod file.

The fetchRawLatestVersion function implements the full definition of
"raw latest" as embodied in the current (go 1.16) go command.  In some
cases this requires looking for the presence of a go.mod file.
Although we could always download the zip from the proxy, the function
accepts a function argument that can avoid that. The worker will pass
a function that looks in the database.

For golang/go#44437

Change-Id: I84e19a4a7a10a87d5664217e0404bc795b3f9470
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295429
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

When fetching a module at a version, the proxy datasource uses
fetch.RawLatestInfo to get the go.mod file at the raw latest version
of the module, then uses internal.RawLatestInfo.PopulateModule to
determine whether the module version is deprecated or retracted.

Also, add some proxy test modules to facilitate testing.

For golang/go#41321
For golang/go#43265
For golang/go#44437

Change-Id: I312346d72f656e598ad170135046ef85da8e9b11
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295430
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

Add a table that will let us track the raw latest version
of each module, and store its go.mod file as well.

For golang/go#44437

Change-Id: I7cf9530a306a87d0c399b326ccffb7033e33461d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295449
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

Add functions to get a row from the raw_latest_versions table,
and to update a row if the new version is later.

For golang/go#44437

Change-Id: I9f44b815b4eb42c6cd286f10f2e946e753c38989
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295450
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

Convert the proxy client to one which doesn't fetch at the beginning
of the request, so we don't have to pass a boolean argument through
several layers.

For golang/go#44437

Change-Id: I0495b5673cc399a1cdc16349d53ba4a9a4ecc380
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295889
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

Check for a row in the paths table before inserting.

For golang/go#44437

Change-Id: I763867c27b1d729311b7ec545c5cdab4e9b64036
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295894
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>

Every time a module is fetched, update its raw latest information in
the database to the most recent value.

This is a simple way of making sure the DB has the most recent raw
latest info.  In the steady state it's unlikely to be wasteful, since
most of the time the new module version will be the latest, so the
update is needed.

For golang/go#44437

Change-Id: If64bca41b51fe63fb2aeb836d6ed98543090cd18
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295890
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>

To make sure it happens in our integration test.

For golang/go#44437

Change-Id: I0e921b067f6798d108880eae152edc5264168e74
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296814
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

Add more information to the latest_module_versions table to help understand
why modules are missing from it.

For golang/go#44437

Change-Id: I7c2b2b2aba2bc76b9302698d3baab4d743fd9eef
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/300909
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

…_versions

For golang/go#44437

Change-Id: I2c655cbdbe0cf622be00acc26a95aa6979a88619
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/300910
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

If we fail to get information about latest module versions, write a
row into the table indicating the failure, rather than not writing
anything at all.

This will let us distinguish missing rows (no attempt) from failed
attempts.

For golang/go#44437

Change-Id: Ibcf16f018c7b5c0805dd511427bfc5ad3d05817e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/300911
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>

A few things can go wrong with computing latest-version information.

Some problems don't necessarily interfere with the calculation, like
the proxy's @latest endpoint returning a "not fetched" error. We can
still use the tagged versions in that case.

Others are more serious (bad go.mod file) but not our fault, so we
shouldn't log them at error level. We want error logs to show only our
bugs.

For golang/go#44437

Change-Id: Id70bfc03ea6fef1de4ef295e0c1d150bca68c883
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/301029
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>