Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/pkgsite: use io/fs in fetch #47834

Closed
jba opened this issue Aug 20, 2021 · 15 comments
Closed

x/pkgsite: use io/fs in fetch #47834

jba opened this issue Aug 20, 2021 · 15 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. pkgsite

Comments

@jba
Copy link
Contributor

jba commented Aug 20, 2021

Go 1.16 introduced the io/fs package, an abstraction over filesystems. Replace internal/fetch's use of achive/zip with io/fs so it can operate on ordinary filesystems, repos, and other implementations of fs.FS as well as zip files.

@jba jba added NeedsFix The path to resolution is known, but the work has not been done. pkgsite labels Aug 20, 2021
@jba jba added this to the pkgsite/unplanned milestone Aug 20, 2021
@jba jba self-assigned this Aug 20, 2021
@gopherbot
Copy link

Change https://golang.org/cl/343951 mentions this issue: internal/fetch: use FS for large forks

@gopherbot
Copy link

Change https://golang.org/cl/343952 mentions this issue: internal/fetch: use fs.FS

@gopherbot
Copy link

Change https://golang.org/cl/343950 mentions this issue: internal/licenses: use fs.FS

@gopherbot
Copy link

Change https://golang.org/cl/343953 mentions this issue: internal/fetch: move all getters to a single source file

@gopherbot
Copy link

Change https://golang.org/cl/343956 mentions this issue: internal/licenses: remove Files and unexport Paths

@gopherbot
Copy link

Change https://golang.org/cl/343958 mentions this issue: internal/licenses: work on content directory

@gopherbot
Copy link

Change https://golang.org/cl/343959 mentions this issue: internal/fetch: use the content directory throughout

@gopherbot
Copy link

Change https://golang.org/cl/343960 mentions this issue: internal/fetch: remove empty-module-path code

@gopherbot
Copy link

Change https://golang.org/cl/343962 mentions this issue: internal/fetch: move fs_test.go into getters_test.go

@gopherbot
Copy link

Change https://golang.org/cl/343964 mentions this issue: internal/fetch: avoid creating zip

@gopherbot
Copy link

Change https://golang.org/cl/343961 mentions this issue: internal/fetch: ModuleGetter.FS returns the content dir

@gopherbot
Copy link

Change https://golang.org/cl/343963 mentions this issue: internal/fetch: rename ModuleGetter.FS to ContentDir

@gopherbot
Copy link

Change https://golang.org/cl/343965 mentions this issue: internal/stdlib: add ContentDir function

@gopherbot
Copy link

Change https://golang.org/cl/343966 mentions this issue: internal/{fetch,worker,stdlib}: remove stdlib.Zip

@gopherbot
Copy link

Change https://golang.org/cl/344389 mentions this issue: internal/fetch: fs getter: check for .zip

gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
Use fs.FS in place of zip.Reader.

Leave existing code for now for backwards compatibility.

For golang/go#47834

Change-Id: Ide39dbef18504e156e70036db67221842088011f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343950
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
Replace ZipSignature with FSSignature.

For golang/go#47834

Change-Id: Ie6cc6053a21260b3e63a1b47359a95731d1be63b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343951
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
Rewrite fetch code to use fs.FS instead of zip.Reader.

For golang/go#47834

Change-Id: Iefdd16b367218690c4e5bea2a4688bea10a94be1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343952
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
Pure code motion.

For golang/go#47834

Change-Id: Ieef2491431fa37b68a184eb9e8e309d3437a8f48
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343953
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
Remove the Files method. It is not used by us,
or anyone else inside Google.

Also, unexport its replacement, the paths method, because it is also
unused outside the module so there is no need to export it.

For golang/go#47834

Change-Id: Iace20d879db753575563067f60ac45e3ea034f31
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343956
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
Change the license detector so it works on the content directory of
the zip, not the root.

Preserve NewDetector's behavior because it's used elsewhere.

For golang/go#47834

Change-Id: I42ba0c95cf19dc735b08a342f48323490d21aa0c
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343958
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
Obtain an fs.FS for the module's content directory
at the beginning of fetching, and use it throughout.

For golang/go#47834

Change-Id: I85684b59a50e06a043affb4d65ce6962c5d23077
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343959
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
An empty module path in fetch won't happen since we made the
DirectoryModuleGetter figure out its module path.

For golang/go#47834

Change-Id: I953ad425c7e13ec4e771f4160870e5ef8f7349ff
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343960
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
Move the responsibility for getting the module zip's content directory
from FetchModule to the ModuleGetters.

This will enable a ModuleGetter that reads directly from the on-disk
filesystem.

For golang/go#47834

Change-Id: I035a88680569272845ad12deb09ed94e6a37bf66
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343961
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
Pure code motion.

For golang/go#47834

Change-Id: I6ccf5ea9574b2d7c9495956ea080cca269c99470
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343962
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
For golang/go#47834

Change-Id: I37db9b43e2ab29a65ec5c5e3860ad859cfb5c7cd
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343963
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
As the culmination of the last few CLs, we no longer need to build a
zip file when reading a module from a local directory.

Now `go run ./cmd/pkgsite .` takes about 12 seconds, instead of too long
to wait for.

For golang/go#47834

Change-Id: I73bdcefe01068432de37c53e3caeebf2b161c9f6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343964
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
Add a function that returns the content directory of the stdlib
module.

Remove the part of the test that checks go.mod; it was never
executing, because there is no go.mod file in any of the repos being
tested.

For golang/go#47834

Change-Id: Idc982620f6736ec60fe9a295f0372fd272745457
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343965
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
For golang/go#47834

Change-Id: I81acf4fdc51a19352bfa7fb0cf9a2c22bb42a65b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343966
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Aug 23, 2021
In the module cache, some directories have .info and .mod files but no
.zip, because the go command doesn't need it. We can't display these
modules without the zip. So return NotFound for these modules, even
for Info and Mod.

Also, return derrors.NotFound any time we can't find a file.

Also, rename fsModuleGetter to fsProxyModuleGetter to make it clear
that the filesystem has to be organized like the proxy protocol.

For golang/go#47834

Change-Id: Ia7d6260abc91a6172820d9de6bf759567720a443
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/344389
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
@jba jba closed this as completed Aug 23, 2021
@rsc rsc unassigned jba Jun 23, 2022
@golang golang locked and limited conversation to collaborators Jun 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. pkgsite
Projects
None yet
Development

No branches or pull requests

2 participants