Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go 'go mod download <package name>' fetches all modules if module cache is empty/missing #40245

Closed
mikemoate opened this issue Jul 16, 2020 · 3 comments
Closed

cmd/go 'go mod download <package name>' fetches all modules if module cache is empty/missing #40245

mikemoate opened this issue Jul 16, 2020 · 3 comments
Labels
FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unplanned

Comments

@mikemoate
Copy link

mikemoate commented Jul 16, 2020 •
edited

What version of Go are you using (go version)?

$ go version

go version go1.14.5 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/hello/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build922582841=/tmp/go-build -gno-record-gcc-switches"

What did you do?

https://golang.org/ref/mod#tmp_28 describes how go mod download can be used to pre-fill the module cache, and how the module paths/patterns to download can be specified. In practice, we find executing this command with an empty/missing module cache causes the download of all modules in go.mod and their dependencies, rather than just the specified module path.

A minimal example to reproduce this (using Docker to ensure a clean Go environment each time) is at https://github.com/mikemoate/go-mod-download-issue the output of this is:

docker build Output
$ docker build .

Sending build context to Docker daemon  71.68kB

Step 1/6 : FROM golang:1.14

---> 2b88645b406c

Step 2/6 : WORKDIR /hello

---> Using cache

---> 59e415a309ac

Step 3/6 : COPY . .

---> f6315f0f3481

Step 4/6 : RUN go mod download -x rsc.io/quote

---> Running in dcf267779b36

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.mod

# get https://proxy.golang.org/github.com/google/uuid/@v/v1.1.1.mod

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.mod: 200 OK (0.127s)

# get https://proxy.golang.org/rsc.io/sampler/@v/v1.3.0.mod

# get https://proxy.golang.org/github.com/google/uuid/@v/v1.1.1.mod: 200 OK (0.141s)

# get https://proxy.golang.org/rsc.io/sampler/@v/v1.3.0.mod: 200 OK (0.013s)

# get https://proxy.golang.org/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.mod

# get https://proxy.golang.org/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.mod: 200 OK (0.012s)

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.info

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.info: 200 OK (0.012s)

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.zip

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.zip: 200 OK (0.013s)

Removing intermediate container dcf267779b36

---> 0a038686c447

Step 5/6 : RUN go version

---> Running in 0d7f569a56eb

go version go1.14.5 linux/amd64

Removing intermediate container 0d7f569a56eb

---> 50c1d1104909

Step 6/6 : RUN go env

---> Running in b613c094cd1e

GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/root/.cache/go-build"

GOENV="/root/.config/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOINSECURE=""

GONOPROXY=""

GONOSUMDB=""

GOOS="linux"

GOPATH="/go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/usr/local/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"

GCCGO="gccgo"

AR="ar"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD="/hello/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build704505364=/tmp/go-build -gno-record-gcc-switches"

Removing intermediate container b613c094cd1e

---> fcdae266e02e

Successfully built fcdae266e02e

go.mod Content
module main

go 1.14


require (

github.com/google/uuid v1.1.1

rsc.io/quote v1.5.2

)

You can clearly see the get for module github.com/google/uuid, despite this not being requested and not being a dependency of the requested module rsc.io/quote

Our use case here is to pre-fetch private modules individually, specifying appropriate credentials for each, ahead of performing a build. This fails because the attempt to fetch an individual private module, tries to download all the private modules in go.mod.

What did you expect to see?

Only the requested module is downloaded.

What did you see instead?

All modules were downloaded.
@mikemoate mikemoate changed the title 'go mod download <package name>' fetches all packages if package cache is empty/missing 'go mod download <package name>' fetches all modules if module cache is empty/missing Jul 16, 2020
@andybons
Copy link
Member

@jayconrod @matloob @bcmills

@andybons andybons added GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 16, 2020
@andybons andybons added this to the Unplanned milestone Jul 16, 2020
@andybons andybons changed the title 'go mod download <package name>' fetches all modules if module cache is empty/missing cmd/go 'go mod download <package name>' fetches all modules if module cache is empty/missing Jul 16, 2020
@bcmills
Copy link
Contributor

bcmills commented Jul 17, 2020

If you go mod download a module without a specific version, the go command uses the version implied by the main module's go.mod file. In order to compute that version, it reads the go.mod file and the go.mod files of the transitive dependencies, because the go.mod file could be in an inconsistent or misleading state and the actual selected version may be higher.

If you were to specify an explicit version for the module, then in theory we would not have to load the dependencies, but we do anyway today. (That is #36812.)

With the proposed lazy loading semantics (#36460), we would make stronger consistency assumptions and load fewer go.mod files (in this case, just the main module's go.mod file, since it specifies the version of the requested module directly). However, in general we may still have to load go.mod files for one layer of transitive dependencies to find the appropriate version.

@bcmills
Copy link
Contributor

bcmills commented Jul 17, 2020

Duplicate of #36812

@bcmills bcmills marked this as a duplicate of #36812 Jul 17, 2020
@bcmills bcmills closed this as completed Jul 17, 2020
@golang golang locked and limited conversation to collaborators Jul 17, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
4 participants
@andybons @bcmills @gopherbot @mikemoate