A new experiment is added, ExperimentUsePathInfoToCheckExistence, which
uses the paths table to check if a package/module/directory exists,
regardless of whether we are using the old or new data model to render
the pages.

This is a first step to use the new paths-based data model. It also
streamlines the flow for frontend fetches.

Updates golang/go#39629

Change-Id: I6178776264cbe71ffb3fb87b3409df902b33b2d4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240680
Reviewed-by: Jonathan Amsterdam <jba@google.com>

GetImports now reads from the package_imports table when the
"use-package-imports" experiment is on.

Updates golang/go#39629

Change-Id: I80970eb4806c8ccf2c01e5dac6a04ff0978fd681
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241181
Reviewed-by: Jonathan Amsterdam <jba@google.com>

Tests in internal/worker now use GetModuleInfo instead of
LegacyGetModuleInfo.

Updates golang/go#39629

Change-Id: I524750809c3e2dcf795bb026636fb76aeba280db
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241064
Reviewed-by: Jonathan Amsterdam <jba@google.com>

The search endpoint now uses GetPathInfo to check if a package
exists, when the "use-path-info" experiment is on.

Updates golang/go#39629

Change-Id: I5c4b3c3250e06a33a647aac74f91849ceb4105ca
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241182
Reviewed-by: Jonathan Amsterdam <jba@google.com>

LatestVersion now uses GetPathInfo to check for the latest version of a
path, when the "use-path-info" experiment is on.

Updates golang/go#39629

Change-Id: I374f707cb06ccd8147925229eb48f2d08caf0a4f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241183
Reviewed-by: Jonathan Amsterdam <jba@google.com>

The insert-directories experiment flag is deleted, since we have already
inserted data for all modules into the paths, package_imports,
documentation, and readmes table, and been running that code path for a
while.

Updates golang/go#39629

Change-Id: I323850a462672c41ad0c67b6ab2b173bb32bf441
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241320
Reviewed-by: Jonathan Amsterdam <jba@google.com>

serveDetailsPage is added, which replaces servePackagePageNew in serving
details pages when the "use-directories" experiment flag is on. Like
servePackagePageNew, serveDetailsPages supports the package/directory
views. It also supports the module view, so that we can stop using
LegacyModuleInfo and reading from modules.readme_file_path and
modules.readme_contents.

For golang/go#39629

Change-Id: I8e664bf1e9174a630db6b723c949ba1e9aa0dc9b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241397
Reviewed-by: Jonathan Amsterdam <jba@google.com>

The two functions for creating a frontend.Package, legacyCreatePackage
and createPackageNew, are merged into a single createPackage function.

createPackage uses the new PackageMeta type, as input. It does not take
any legacy structs as input and returns a frontend.Package.

In future CLs, PackageMeta will be used to LegacyDirectory, when
fetching data for the directories tab.

For golang/go#39629

Change-Id: I80ec5272c6f8e237f0752938a89c633e3a1b81f5
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241441
Reviewed-by: Jonathan Amsterdam <jba@google.com>

…s functions

The GetPsuedoVersions* and GetTaggedVersions* functions read from the
packages table, and will be replaced by a single GetVersionsForPath
function.

These functions are now prefixed with "Legacy" to indicate that they
will be deprecated.

For golang/go#39629

Change-Id: I7f89f9890f135b5ddb363a51a9706e48d02594b5
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241899
Reviewed-by: Jonathan Amsterdam <jba@google.com>

The licenses table currently has two FKs to modules.
licenses_module_path_fkey is no longer needed and is dropped.

For golang/go#39629

Change-Id: Ibf5a6400b1cf90e82748dcc53f45d23fdc29286d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/265077
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>

A unique index is added to licenses(module_id, file_path).

This is temporary to enable insertLicenses, until the PK is switched.

For golang/go#39629

Change-Id: Iba9d5da93e8dffe477cada454a84248bd48a9b3c
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/265240
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>

insertLicenses now uses licenses (module_id, file_path) as the ON
CONFLICT key.

For golang/go#39629

Change-Id: Ib85a61f538a176c1bfff7a070121162405ef21dc
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/265241
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>

The primary key for licenses is changed to (module_id, file_path).

The licenses.module_path and licenses.version columns will be dropped in
a future CL.

For golang/go#39629

Change-Id: I3ae8f7c582013a897152bdf5926f6a9287136c5e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/265019
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>

The columns modules.readme_file_path and modules.readme_contents are
dropped, since they are no longer being used.

For golang/go#39629

Change-Id: I3829bdeb4eea9a5b8fb4a706641714a3aa8bfefc
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/265097
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>

TestFetchAndUpdateState_NotFound no longer references the imports table.

For golang/go#39629

Change-Id: Icb60c0bf46656cb009c84302778ccf7f8e1fcbce
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/265249
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>

The imports table is no longer used and is dropped.

This frees up 136 GB of table space and 478 GB of indexes.

For golang/go#39629

Change-Id: I5a381832d9d4d69408fb62052782097436c40371
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/265242
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>

The packages table is no longer used and dropped.

This frees up 827 GB of table space and 78 GB of indexes.

For golang/go#39629

Change-Id: I149707013c3fd39122b55f79976893ca979982cb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/265243
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>

idx_paths_path is a duplicate of index paths_path_module_id_key and is dropped.

This frees up 16 GB of index space.

For golang/go#39629

Change-Id: Ia6baa1c47a00c2724ab85274787d68befc5ac92d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/265247
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>

A search_documents.path_id column is added, which will become a FK to
the paths table.

For golang/go#39629

Change-Id: I4c24c1f93229afb758d4ca0a451fd3a4f6d01f0b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/265248
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>

A paths table is added, which contains the path string for every path in
the units table.

This table is added to replace units.path in order to reduce the index
size on that column and improve performance.

For golang/go#39629

Change-Id: I9d173bbb8a1680176f1807d8d3561cd6906afaf3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/267437
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>

getModuleLicenses now accepts moduleID instead of module path and
version as args.

In a later CL, we will be dropping licenses.module_path and
licenses.module_id.

For golang/go#39629

Change-Id: I18c525c584fd181372ef6c01241a90c19cb28a96
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/271747
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>

Drop the NOT NULL constraint on licenses.module_path and
licenses.version, so that we can stop inserting into these
rows in the next CL.

In a later CL, licenses.module_path and licenses.version will be
dropped.

For golang/go#39629

Change-Id: Ie917498306d46e03a761af32406356774a18fc78
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/272086
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>

…version

In a later CL, licenses.module_path and licenses.version will be
dropped.

For golang/go#39629

Change-Id: I7de03b66686e04128c761f477130eb98571a8cd2
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/272087
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>