Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proposal: cmd/go: add GOPROXY=cache to rely only on module cache #43646

Open
Merovius opened this issue Jan 12, 2021 · 10 comments
Open

proposal: cmd/go: add GOPROXY=cache to rely only on module cache #43646

Merovius opened this issue Jan 12, 2021 · 10 comments
Labels
Proposal
Projects
Proposals (old)
Milestone
Proposal

Comments

@Merovius
Copy link
Contributor

When I try to import a yet-unrequired module, e.g.

mero@hix ~/src/github.com/Merovius/foo$ cat foo.go 
package main

import "rsc.io/qr"

func main() {
	var _ qr.Code
}
mero@hix ~/src/github.com/Merovius/foo$ cat go.mod
module github.com/Merovius/foo

go 1.15

and I run go mod tidy, while not connected to the internet, I get an error

go: finding module for package rsc.io/qr
github.com/Merovius/foo imports
	rsc.io/qr: module rsc.io/qr: Get "https://proxy.golang.org/rsc.io/qr/@v/list": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:57927->[::1]:53: read: connection refused

That makes sense - the proxy is not reachable. However, my module cache contains a usable version of this module:

mero@hix ~/src/github.com/Merovius/foo$ l -d ~/pkg/mod/rsc.io/qr@v0.2.0 
dr-x------. 5 mero mero 4,0K 14. Mai 2020  /home/mero/pkg/mod/rsc.io/qr@v0.2.0

So, it might be a reasonable course of action for me, to just use the version I have in the cache - whether it is the latest, or not. If I want to do that, as far as I can tell, either

a) I have to look up what versions I have in my cache and run go get rsc.io/qr@v0.2.0, to update go.mod. It would be great, if that could be automated, or
b) set GOPROXY=file:///home/mero/pkg/mod/cache/download and run go mod tidy

Both of these can be considered somewhat clunky (and the latter is not very discoverable). I would like to suggest to make this easier. There are several ways this could happen:

  1. Automatically fall back to the module cache when running go mod tidy or similar and a network error happens. IMO that would be the most convenient - it also has the downside of unexpectedly adding an outdated version of the module.
  2. Add a value to the -mod flag that is equivalent to -mod=mod, but only touching the local cache.
  3. Add a flag to go mod tidy (and similar), to use the local cache as a proxy (essentially automate setting the environment variable).

It's also fine, if this is deemed WAI, of course. I just thought I would suggest it after the case came up on golang-nuts.
@gopherbot gopherbot added this to the Proposal milestone Jan 12, 2021
@jayconrod
Copy link
Contributor

cc @bcmills @matloob @rsc

Closely related: #32337

We've also talked about adding GOPROXY=cache, which would be similar to GOPROXY=file://$(go env GOMODCACHE)/cache/download, but would synthesize version lists and .info files.

@Merovius
Copy link
Contributor Author

GOPROXY=cache would also address this and is, I think, a very logical place for that knob (and far easier to remember than the path).

@ianlancetaylor ianlancetaylor changed the title Proposal: cmd/go: Fallback to module cache, if proxy is not reachable proposal: cmd/go: fallback to module cache, if proxy is not reachable Jan 12, 2021
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Jan 12, 2021
@rsc rsc changed the title proposal: cmd/go: fallback to module cache, if proxy is not reachable proposal: cmd/go: add GOPROXY=cache to rely only on module cache Jul 15, 2021
@bcmills bcmills mentioned this issue Aug 9, 2021
Open
@bcmills bcmills mentioned this issue Nov 5, 2021
Closed
@mvdan
Copy link
Member

mvdan commented Nov 5, 2021

Just so others find this issue more easily via keywords on google: I was using "offline mode" in my search.

@bcmills bcmills mentioned this issue Mar 2, 2022
Open
dolmen added a commit to dolmen-go/goeval that referenced this issue Feb 7, 2023
@dolmen
Use Go cache if modules are in found in the cache
b8e0d7d 
In module mode, when all module dependencies are given with canonical
versions and the version is found in the cache, set GOPROXY to
file://$(go env GOMODCACHE)/cache/download so that "go get" skips
querying the proxy.

Related: golang/go#43646 (proposal for
GOPROXY=cache)
@dolmen dolmen mentioned this issue Feb 7, 2023
Merged
@dolmen
Copy link
Contributor

dolmen commented Feb 7, 2023

Use case: for goeval (Go one-liners), when the same command is run multiple times, querying the proxy every time while everything is available locally is a waste of resources and has a huge performance impact (longer run time) for the user (note: I also understand that this is a security feature as it allows to block retracted versions).

So, as a workaround for that go get slowness I've implemented a shortcut: set GOPROXY=file://$(go env GOMODCACHE)/cache/download if all module versions are canonical and found in the cache.

However, it would be appreciated if go get had an built-in opt-in mode where dependencies would be resolved locally if possible, and the GOPROXY would be still be used but only as a fallback. Because my shortcut is limited: it is all or nothing: if just one version is non-canonical (ex: @latest), all dependencies resolution will hit the proxy.

dolmen added a commit to dolmen-go/goeval that referenced this issue Feb 8, 2023
@dolmen
Use Go cache if modules are in found in the cache
2557dd1 
In module mode, when all module dependencies are given with canonical
versions and the version is found in the cache, set GOPROXY to
file://$(go env GOMODCACHE)/cache/download so that "go get" skips
querying the proxy.

Related: golang/go#43646 (proposal for
GOPROXY=cache)
@dolmen
Copy link
Contributor

dolmen commented Mar 16, 2023 •
edited

GOPROXY=off is documented at https://go.dev/ref/mod#environment-variables.

Does it fit the need from this proposal?

@mvdan
Copy link
Member

mvdan commented Mar 16, 2023

@dolmen I don't think so, because multiple commands try to resolve @latest and that will often still try to use the network to fetch source from upstream's VCS with GOPROXY=off.

@dolmen
Copy link
Contributor

dolmen commented Mar 16, 2023

The documentation about resolving packages to a module claims:

off indicates that no communication should be attempted.

@mvdan
Copy link
Member

mvdan commented Mar 16, 2023

Right, I got slightly confused with GOPROXY=direct. But I still think that GOPROXY=off is not enough for an "offline mode" to work well enough with what's in the module cache. For example, go get module@latest simply fails with module lookup disabled by GOPROXY=off rather than finding the latest version that's already in the module cache.

@dolmen
Copy link
Contributor

dolmen commented Mar 16, 2023

Ok, so in GOPROXY=cache mode $module@latest would be resolved using content of $GOMODCACHE/cache/download/$module/@v/list, isn't it?

@mvdan
Copy link
Member

mvdan commented Mar 16, 2023

That's how I would imagine it, yes, and it's what I understood from #43646 (comment). It's worth noting that "latest" might be out of date in such a cache/offline mode, but that's by design.

@dolmen dolmen mentioned this issue Mar 16, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Proposal
Projects
Proposals
Status: Incoming
Proposals (old)
Incoming
Milestone
Proposal
Development

No branches or pull requests
5 participants
@dolmen @jayconrod @Merovius @mvdan @gopherbot