x/pkgsite: automatically fetch a requested package if it does not exist on pkg.go.dev #37002

natefinch · 2020-02-03T21:23:04Z

Currently, if you request package docs from pkg.go.dev for a package that hasn't been sent to the go proxy yet, then it just returns 404.

Requesting a url from pkg.go.dev should do whatever "add your package" does, if that's required to view the docs, because clearly I want to view the docs and the server knows exactly what step is missing to allow me to view the docs, so just do it, please.

By returning a 404 page, it looks like there's a bug in the website. The suggested "how to add your package" requires some arcane steps (I presume? it's not actually spelled out) that simply shouldn't be necessary.

See #36986 for an example of someone reporting a "bug" because a package didn't show up.

This is actually super common when using godoc.org in my experience, as I will often want to view the docs on godoc.org as I'm writing the code, to see how they look there. I'm sure others will want to do the same on pkg.go.dev.

This can also easily happen if most people in your company use a corporate proxy, so the code isn't in go's proxy yet, but then you decide you want to view the docs and you're just used to doing it on pkg.go.dev (which I do all the time for random go packages on godoc.org... rather than download them and view the docs locally, I view 99% of docs directly on godoc.org).

hunterloftis · 2020-02-07T15:54:40Z

I remember the "magic" of having my open-source projects auto-doc'd for me. Someone had to point out to me that I no longer needed to build my own markdown-generating toolchain for docs - instead, they would "just exist" and even be searchable, alongside major, important packages.

I believe this suggestion aligns with that beginner-friendly "it just works" philosophy that has long been a part of the language and community.

ghost · 2020-02-08T02:30:27Z

If immediate fetch is not possible for some reason, then add a "request documentation" button to the 404 page. Clicking the button kicks of whatever needs to be done and shows UI explaining when the documentation will be available.

deeper-x · 2020-02-27T15:07:11Z

I'm joining this thread because I consider this issue a detrimental change in the language.
The way godoc.org works is how things should be, as Andrew Gerrand says

[...] it should be coupled to the code itself so the documentation evolves along with the code. The easier it is for programmers to produce good documentation, the better for everyone

Plain and simple: in Go, code & documentation are tightly coupled by default. No surprises expected, because publishing your code you're publishing the documentation on godoc.org as well.

What you quietly introduced with pkg.go.dev and mention there breaks this natural flow introducing a weird exception, silently assuming that now could exist public code-without-documentation.
Furthermore, suggesting a workaround like forcing a fake download in order to trigger the documentation generation, cannot be considered a stable solution.

The logic for GetVersionMap is changed so that an explicit module path is now required. All logic is also now the same for all requestedVersions (as opposed to a special case for latest). This function is only used by the frontend to check if a version exists as part of a frontend fetch, and this behavior makes more sense for that use case. Updates golang/go#36811 Updates golang/go#37002 Updates golang/go#37106 Change-Id: I415a2730daa6edc023f80c0c615521047311f35b Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/744833 Reviewed-by: Jonathan Amsterdam <jba@google.com>

The frontend server is not initiated with a queue. The frontend task queue will be used to support frontend fetches. frontend.FetchAndUpdateState is added, which is a copy of worker.FetchAndUpdateState for use in testing and locally. Updates golang/go#36811 Updates golang/go#37002 Updates golang/go#37106 Change-Id: I41922d30462d2623a061aa1f207bb2b39f7b54e2 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/743102 Reviewed-by: Jonathan Amsterdam <jba@google.com>

This column will be used to display the canonical module path when a frontend fetch results in fetching an alternative module path. Updates golang/go#36811 Updates golang/go#37002 Updates golang/go#37106 Change-Id: Ie8ee6ba64ea799d264c1079c345e37d2e073da38 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/751268 Reviewed-by: Jonathan Amsterdam <jba@google.com>

The version_map.go_mod_path column is now populated in UpsertVersionMap. Updates golang/go#36811 Updates golang/go#37002 Updates golang/go#37106 Change-Id: I4e474d28bee67caf7625045082df46f93394f3f6 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/751269 Reviewed-by: Jonathan Amsterdam <jba@google.com>

FetchAndUpdateState in internal/frontend and internal/worker now insert the GoModPath field on VersionMap. Updates golang/go#36811 Updates golang/go#37002 Updates golang/go#37106 Change-Id: Idfce6d685d24d915608acc1748a84b9db5312ae4 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/751270 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>

A fetch endpoint is added to the frontend, which will queue a module to be enqueued by the worker, if it doesn't already exist in the database. After enqueuing, the fetch handler will poll the version_map table at a constant rate, until the path returns or the request times out. If the request fails, a corresponding statusCode and responseText will be returned to be displayed to the user. Updates golang/go#36811 Updates golang/go#37002 Updates golang/go#37106 Change-Id: Ic2e20146dc626bf296db05bc2abbfb50d6fd7991 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/743103 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>

When a path page 404s, it will now render the notfound.tmpl page (once the frontend-fetch feature flag is on), which provides a button for the user to make a request to fetch the package. Updates golang/go#36811 Updates golang/go#37002 Updates golang/go#37106 Change-Id: I17fedd018435e8d3e51e2a2a4a972d3cf673df56 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/753606 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>

At the moment when a request for path@version 404s, but other versions of the path exists, we will return an error message letting users know that is the case. Once frontend fetch is live, the message will change to provide users an option to fetch that version of the path. This logic is moved to pathFoundAtLatestError. Updates golang/go#36811 Updates golang/go#37002 Updates golang/go#37106 Change-Id: I1ad15ee13714a68b4b88fd353e43719fda0c0d31 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/754818 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>

When a fetch request results in a 404, it's possible that the module path will appear later in the index. Similarly, when a fetch request results in a 480 or 500, it's possible that the error was transient or has been fixed but not yet reprocessed. In these cases, we should allow the user to refetch the path after the taskIDChangeInterval has passed. For golang/go#36811 For golang/go#37002 Change-Id: Ia4813140ca793b7b91d80bdc196f61af075766ca Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244601 Reviewed-by: Jonathan Amsterdam <jba@google.com>

gopherbot · 2020-07-29T03:12:47Z

Change https://golang.org/cl/245437 mentions this issue: content: alphabetize classes with prefix Fetch

gopherbot · 2020-07-29T03:12:49Z

Change https://golang.org/cl/245402 mentions this issue: internal/worker: don't delete 40x results from version_map

gopherbot · 2020-07-29T03:12:51Z

Change https://golang.org/cl/245401 mentions this issue: content: center text in h3.FetchMessage

Class names with the prefix "Fetch" are alphabetized relative to each other. No changes are made to the CSS. For golang/go#36811 For golang/go#37002 Change-Id: I662829fd50f1e1634cc891ba50c4bf1d4f9ac0bd Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245437 Reviewed-by: Jonathan Amsterdam <jba@google.com>

At the moment, when module is fetched with a status 40x, it is deleted from the database. However, we don't want to delete that module from version_map, otherwise the response will never be returned to the user for a frontend fetch. Modules with status 40x are now deleted first, and then the result is inserted into version_map and module_version_states. For golang/go#36811 For golang/go#37002 Change-Id: I5eb9414c065bdc8364445edb293f21bbe870fa43 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245402 Reviewed-by: Jonathan Amsterdam <jba@google.com>

For golang/go#36811 For golang/go#37002 Change-Id: I3438a8d72617f3c7af88310469b4799a5539616e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245401 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>

gopherbot · 2020-07-29T21:21:05Z

Change https://golang.org/cl/245640 mentions this issue: internal/frontend: tweak response text for fetch requests

The response text for different frontend fetch outcomes is updated. The code in fetchRequestStatusAndResponseText is also refactored. For golang/go#36811 For golang/go#37002 Change-Id: I175921acde5edc5cbf677eb1d1f23171b40318b9 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245640 Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>

gopherbot · 2020-07-29T23:50:48Z

Change https://golang.org/cl/245646 mentions this issue: content: update text in fetch.tmpl for 404s

For golang/go#36811 For golang/go#37002 Change-Id: I1853ecf2798e717589bb81ad0d749a3f814b55ec Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245646 Reviewed-by: Jonathan Amsterdam <jba@google.com>

gopherbot · 2020-07-30T16:03:40Z

Change https://golang.org/cl/245880 mentions this issue: internal/fronted: update text in fetch.tmpl

gopherbot · 2020-07-30T16:04:18Z

Change https://golang.org/cl/245878 mentions this issue: internal/frontend: return 404 on invalid stdlib paths

The text in fetch.tmpl is updated to provide clearer instructions. Corresponding CSS is also adjusted for these text changes. For golang/go#37002 Change-Id: Iae8b641eb2471172eb4465493fe83a4005bce6fc Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245880 Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jamal Carvalho <jamal@golang.org>

At the moment, pkg.go.dev/404 returns the following error message: Package “404” could not be found, but you can view module “std” at https://pkg.go.dev/mod/std. This is because stdlib.Contains(404) returns true. This error is fixed so that pkg.go.dev/404 just shows a normal 404 error, when frontend fetch is active. For golang/go#37002 Change-Id: I787a10009294d105d3f92df19790dcd438123f8d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245878 Reviewed-by: Jonathan Amsterdam <jba@google.com>

gopherbot · 2020-07-30T17:24:21Z

Change https://golang.org/cl/245897 mentions this issue: internal/frontend: improve validation of candidate module paths for fetch

…etch Additional source hosts are added to pre-filter invalid module paths. Each path is also validated using module.CheckImportPath. For golang/go#37002 Change-Id: I4090a915c023b4d84b9b139e74256c61ca183176 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245897 Reviewed-by: Jonathan Amsterdam <jba@google.com>

julieqiu · 2020-07-30T20:03:20Z

Users can now request a package by clicking a button on the 404 page.

Here's an example:

criscola · 2021-01-21T12:43:18Z

Users can now request a package by clicking a button on the 404 page.

Is there any way to retrigger the request for a module? I've updated my module (same version, forgot small addition, no users are relying on my package) but I still see the older version.

julieqiu · 2021-01-21T14:30:17Z

@criscola you can either visit https://pkg.go.dev/<your-module-path>@master, or follow the instructions at https://go.dev/about#adding-a-package.

criscola · 2021-01-24T12:55:11Z

@julieqiu I've tried but my godocs is still not getting updated to the latest commit.

julieqiu · 2021-01-25T03:55:30Z

@criscola - since this is a closed issue, could you please file a new issue at https://golang.org/s/pkgsite-feedback?

#43096 also seems related.

gopherbot added the Documentation label Feb 3, 2020

julieqiu added FeatureRequest and removed Documentation labels Feb 3, 2020

gopherbot added the Documentation label Feb 3, 2020

julieqiu added pkgsite NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed Documentation labels Feb 3, 2020

julieqiu changed the title ~~go.dev: requesting unpopulated docs should generate them~~ go.dev: automatically fetch a requested package if it does not exist on pkg.go.dev Feb 4, 2020

gopherbot added this to the Unreleased milestone Feb 6, 2020

dmitshur mentioned this issue Feb 28, 2020

x/pkgsite: automate fetching of new versions of Go standard library #37568

Open

julieqiu mentioned this issue Feb 28, 2020

x/pkgsite: support Refresh now like godoc.org #37106

Closed

julieqiu mentioned this issue Mar 11, 2020

go.dev: search can show "zero results" on an exact package match #37111

Closed

julieqiu mentioned this issue Mar 26, 2020

go.dev: go.dev/about instructions to add package to the docs need clarification #37005

Closed

dmitshur mentioned this issue Apr 9, 2020

go.dev: pkg.go.dev should work for e.g. git.sr.ht #38326

Closed

julieqiu mentioned this issue May 19, 2020

x/pkgsite: godoc.org redirect feedback - tracking issue #39144

Closed

14 tasks

aclements mentioned this issue Jun 1, 2020

go.dev: easy way to update packages #39336

Closed

This was referenced Jun 9, 2020

go.dev: Package version not showing up. #39486

Closed

go.dev: No way to trigger pkg.go.dev update on Package that *was* missing a license #39564

Closed

julieqiu changed the title ~~go.dev: automatically fetch a requested package if it does not exist on pkg.go.dev~~ x/pkgsite: automatically fetch a requested package if it does not exist on pkg.go.dev Jun 15, 2020

julieqiu modified the milestones: Unreleased, pkgsite/godoc.org-redirect Jul 29, 2020

julieqiu closed this as completed Jul 30, 2020

julieqiu mentioned this issue Aug 6, 2020

go.dev: allow force-refresh of package just like on godoc.org #38848

Closed

bang9211 mentioned this issue Oct 6, 2021

x/pkgsite: refresh of pkg.go.dev #48832

Closed

golang locked and limited conversation to collaborators Jan 25, 2022

gopherbot added the FrozenDueToAge label Jan 25, 2022

rsc unassigned julieqiu Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/pkgsite: automatically fetch a requested package if it does not exist on pkg.go.dev #37002

x/pkgsite: automatically fetch a requested package if it does not exist on pkg.go.dev #37002

natefinch commented Feb 3, 2020

hunterloftis commented Feb 7, 2020

ghost commented Feb 8, 2020

deeper-x commented Feb 27, 2020 •

edited

gopherbot commented Jul 29, 2020

gopherbot commented Jul 29, 2020

gopherbot commented Jul 29, 2020

gopherbot commented Jul 29, 2020

gopherbot commented Jul 29, 2020

gopherbot commented Jul 30, 2020

gopherbot commented Jul 30, 2020

gopherbot commented Jul 30, 2020

julieqiu commented Jul 30, 2020

criscola commented Jan 21, 2021 •

edited

julieqiu commented Jan 21, 2021 •

edited

criscola commented Jan 24, 2021

julieqiu commented Jan 25, 2021

x/pkgsite: automatically fetch a requested package if it does not exist on pkg.go.dev #37002

x/pkgsite: automatically fetch a requested package if it does not exist on pkg.go.dev #37002

Comments

natefinch commented Feb 3, 2020

hunterloftis commented Feb 7, 2020

ghost commented Feb 8, 2020

deeper-x commented Feb 27, 2020 • edited

gopherbot commented Jul 29, 2020

gopherbot commented Jul 29, 2020

gopherbot commented Jul 29, 2020

gopherbot commented Jul 29, 2020

gopherbot commented Jul 29, 2020

gopherbot commented Jul 30, 2020

gopherbot commented Jul 30, 2020

gopherbot commented Jul 30, 2020

julieqiu commented Jul 30, 2020

criscola commented Jan 21, 2021 • edited

julieqiu commented Jan 21, 2021 • edited

criscola commented Jan 24, 2021

julieqiu commented Jan 25, 2021

deeper-x commented Feb 27, 2020 •

edited

criscola commented Jan 21, 2021 •

edited

julieqiu commented Jan 21, 2021 •

edited