Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go mod download and go mod tidy produce different go.sum #35832

Closed
roffjulie opened this issue Nov 25, 2019 · 7 comments
Closed

cmd/go: go mod download and go mod tidy produce different go.sum #35832

roffjulie opened this issue Nov 25, 2019 · 7 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@roffjulie
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.13.4 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/juliehockett/.cache/go-build"
GOENV="/usr/local/google/home/juliehockett/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/google/home/juliehockett/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/google-golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/usr/local/google/home/juliehockett/random/go/gosum/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build343345367=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Running 'go mod download' produces a different 'go.sum' file than running 'go mod tidy'.

Given a simple main program with the following 'go.mod':

module example.com/gosum

go 1.13

require rsc.io/quote v1.5.2

Running 'go mod download' produces the following 'go.sum':

golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=

While running 'go mod tidy' produces:

golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c h1:qgOY6WgZOaTkIIMiVjBQcw93ERBE4m30iBm00nkL0i8=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3Y=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
rsc.io/sampler v1.3.0 h1:7uVkIFmeBqHfdjD+gZwtXXI+RODJ2Wc4O7MPEh/QiW4=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=

What did you expect to see?

The same go.sum file produced for both commands.

What did you see instead?

A second line for each dependency in the file produced by 'go mod tidy'. Seems like it's hashing the go.mod file and the package itself, but that's unclear to me.

@ghost
Copy link

ghost commented Nov 26, 2019

Also #27633

@smasher164 smasher164 changed the title go mod download and go mod tidy produce different go.sum files cmd/go: go mod download and go mod tidy produce different go.sum Nov 26, 2019
@toothrot
Copy link
Contributor

@juliehockett Thanks for the detailed example! I agree that this sounds like a duplicate of #27633.

There are some more details here https://github.com/golang/go/wiki/Modules#why-does-go-mod-tidy-record-indirect-and-test-dependencies-in-my-gomod and https://tip.golang.org/cmd/go/#hdr-Maintaining_module_requirements.

I'll close this bug in favor of discussing on #27633.

@roffjulie
Copy link
Author

Thanks for the response! I'm actually thinking this might be subtly different, because we've also run into this as because go mod download is insufficient to build with go build -mod=readonly. The go build command generates the same go.sum file as the go mod tidy command above, and so it errors out under the readonly flag because this difference is persistent there. Is that intended behavior?

Our solution thus far has been to run go mod download && go mod tidy and then do the readonly build, but that's not ideal from a build aesthetic point of view.

(I'm commenting on this bug because the duplicate liked looks like it's talking about differences in the go mod tidy and go build output, not the differences in go mod download and go build, but I could just have an incomplete picture)

@toothrot
Copy link
Contributor

Thanks, I'll re-open. Let's cc @bcmills and @jayconrod who own cmd/go who can help clarify this behavior.

@toothrot toothrot reopened this Nov 26, 2019
@toothrot toothrot added this to the Backlog milestone Nov 26, 2019
@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 26, 2019
@bcmills
Copy link
Contributor

bcmills commented Nov 26, 2019

This is likely related to #29772.

That said, it would be helpful to have an exact reproducer for this: the go.mod file alone is not sufficient, because go mod tidy follows import statements from all of the .go source files in the main module.

@juliehockett, could you provide the .go source files for this example?

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 26, 2019
@roffjulie
Copy link
Author

You bet -- main.go (a simple HTTP test) is:

package main

import (
	"fmt"
	"net/http"
	"os"

	"rsc.io/quote"
)

func handler(w http.ResponseWriter, r *http.Request) {
	if quote.Hello() == "Hello, world." {
		fmt.Fprintf(w, "PASS")
	} else {
		fmt.Fprintln(w, "FAIL")
	}
}

func main() {
	http.HandleFunc("/", handler)
	http.ListenAndServe(":"+os.Getenv("PORT"), nil)
}

@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 26, 2019
@bcmills
Copy link
Contributor

bcmills commented Jan 7, 2020

Oh, I get it now.

go mod download is downloading all of the modules in the dependency graph, which it can determine from reading only the go.mod files. It doesn't know which of those modules are actually needed to satisfy a build, so it doesn't pull in the checksums for those modules (because they may not be relevant).

On the other hand, go mod tidy has to walk the package graph in order to ensure that all imports are satisfied. So it knows, for a fact, without doing any extra work, that the modules it walks through are needed for a go build in some configuration.

So this is a bit confusing (and we should think about that some more when other issues are reported), but the behavior originally reported seems correct, at least in the sense that everything is working as designed and neither command is doing any work beyond what it was asked to do.

For your use case, go mod tidy alone should probably suffice.

@bcmills bcmills closed this as completed Jan 7, 2020
@golang golang locked and limited conversation to collaborators Jan 6, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants