cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u' #29702

SamWhited · 2019-01-12T00:26:02Z

I thought I remembered an issue about this being fixed already, but I couldn't find it and it's still happening and causing problems. Sorry of this is noise and the issue is still floating around somewhere.

The specific dependency in this example that's being stripped out and re-added does not use modules, but does use semantic versioned tags which testify is pinning to.

What version of Go are you using (`go version`)?

$ go version
go version devel +a2bb68de4d1c 20190111160859 linux/amd64

Does this issue reproduce with the latest release?

It happens with go1.12beta2, go1.12beta1, and go1.11.4.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GOARCH="amd64"
GOBIN="/home/sam/bin"
GOCACHE="/home/sam/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/sam/go"
GOPROXY=""
GORACE=""
GOROOT="/home/sam/Projects/go"
GOTMPDIR=""
GOTOOLDIR="/home/sam/Projects/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/sam/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build393255074=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Create a program that imports a package that exhibits this behavior. I noticed it with the Stripe SDK, and testify seems to be the dependency causing the problem so the following also has the same issue:

package main

import (
	"fmt"

	"github.com/stretchr/testify/assert"
)

func main() {
	_ = assert.CallerInfo
	fmt.Println("vim-go")
}

Now run go mod init testmod, and go get -u. You will end up with the following go.mod file:

module testmod

go 1.12

require (
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/stretchr/objx v0.1.1 // indirect
	github.com/stretchr/testify v1.3.0
)

now run go mod tidy and you wind up with the following:

module testmod

go 1.12

require (
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/stretchr/testify v1.3.0
)

What did you expect to see?

The file does not change.

What did you see instead?

Every time you run go mod tidy it strips out indirect dependencies inserted by go get -u and you end up with unmerged changes in your repo.

The text was updated successfully, but these errors were encountered:

mvdan · 2019-01-12T12:46:32Z

/cc @bcmills

bcmills · 2019-01-14T18:12:23Z

Looks like they agree on the build list but disagree about whether dependencies are indirect:

$ go get -u
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/stretchr/objx v0.1.1
go: downloading github.com/davecgh/go-spew v1.1.1

$ cat go.mod
module golang.org/issue/scratch

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/stretchr/objx v0.1.1 // indirect
        github.com/stretchr/testify v1.3.0
)

$ go list -m all
golang.org/issue/scratch
github.com/davecgh/go-spew v1.1.1
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.1
github.com/stretchr/testify v1.3.0

$ go mod tidy

$ cat go.mod
module golang.org/issue/scratch

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/stretchr/testify v1.3.0
)

$ go list -m all
golang.org/issue/scratch
github.com/davecgh/go-spew v1.1.1
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.0
github.com/stretchr/testify v1.3.0

bcmills · 2019-01-14T18:13:03Z

Wait, no. go mod tidy ends up downgrading; that might be a bug.

bcmills · 2019-01-14T19:15:35Z

This case is interesting, though: nothing in the build actually imports from objx, and nothing in objx affects the versions of other libraries, so perhaps go mod tidy is correct to remove it from the go.mod file even though it remains in the transitive module graph.

SamWhited · 2019-01-14T19:23:23Z

This case is interesting, though: nothing in the build actually imports from objx.

The tests do though (via testify/mock), and it appears that mod tidy is adding things behind build tags and test files generally; it's just this one that it's stripping out for some reason.
Even though it doesn't affect any other libraries, if we require pinning to a specific version of objx stripping it out might mean that we pull in a newer version that's broken, no?

bcmills · 2019-01-14T19:36:23Z

The tests do though (via testify/mock),

Nothing in the import graph imports testify/mock. go mod tidy only considers the packages transitively imported by the main module and the transitive dependencies of the tests of those packages.

if we require pinning to a specific version of objx stripping it out might mean that we pull in a newer version that's broken, no?

In this case, go test all literally doesn't build anything in objx, so the only way it could be broken is through some invalid module requirement.

SamWhited · 2019-01-14T19:59:17Z

Nothing in the import graph imports testify/mock. go mod tidy only considers the packages transitively imported by the main module and the transitive dependencies of the tests of those packages.

doc.go imports testify/http which imports testify/mock in a test_round_tripper.go, unless I'm still misunderstanding. It's a weird _ import though. I don't really understand why, not having ever used testify except transitively. It doesn't appear to have any import side effects (thank goodness).

In this case, go test all literally doesn't build anything in objx, so the only way it could be broken is through some invalid module requirement.

I see; I suppose if it's an _ import and nothing is actually used that's fair.

bcmills · 2019-01-14T20:28:41Z

doc.go imports testify/http which imports testify/mock

Nothing in the example program imports github.com/stretchr/testify (that is, the package containing doc.go) itself.

SamWhited · 2019-01-14T20:48:23Z

oops, sorry, you're right of course.

rsc · 2019-01-24T16:39:05Z

Why did go get -u decide to look at objx?

thanasik · 2019-04-03T23:21:42Z

Having this issue as well, but with local private dependencies. Both go mod tidy and go mod vendor alter my go mod file this way, and running go build on the modified go.mod file results in an error during build saying cannot find module for path (though it does actually vendor it, the files are in the vendor dir). I have to manually add the dependency back in order to get the build working. For context, my go version is go version go1.12.1 linux/amd64

bcmills · 2019-04-09T14:49:15Z

This is probably a side-effect of #26902.

thepudds · 2019-05-05T18:54:11Z

This is probably a side-effect of #26902.

Now that #26902 is resolved, I tried to reproduce this using tip (e.g., go get golang.org/dl/gotip && gotip download).

As hoped or expected, github.com/stretchr/objx now no longer shows up after the initial go get -u, and then running go mod tidy has no impact:

gotip mod init testmod
cat <<EOF > main.go
package main
import (
    "fmt"
    "github.com/stretchr/testify/assert"
)
func main() {
    _ = assert.CallerInfo
    fmt.Println("vim-go")
}
EOF
gotip get -u
cp go.mod go.mod_orig
gotip mod tidy
diff go.mod_orig go.mod
cat go.mod

which results in no diff reported, and go.mod contains:

module testmod

go 1.13

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/stretchr/testify v1.3.0
)

I used go version devel +ba9bc8e5 Sun May 5 12:55:21 2019 +0000.

Also, gotip get -u all seems to give same result as gotip get -u (which seems expected as well).

I did not however attempt to evaluate all the commentary in this issue -- I only tried the first set of steps reported here (in comment #29702 (comment)).

@SamWhited would you be willing to try this with tip? Does this seem resolved from your perspective, or does it seem there are still pieces pending?

gopherbot · 2019-06-06T21:04:38Z

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

colega · 2019-07-02T12:37:08Z

This still happens on go version go1.12.6 darwin/amd64 and go version devel +d410642 Mon Jul 1 21:30:23 2019 +0000 darwin/amd64 (gotip).

We're reproducing it in the following scenario, consider two libraries github.com/zero-nine-three and github.com/one-zero-zero that import nothing but github.com/prometheus/client_golang on versions v0.9.3 and v1.0.0 respectively:

$ tail -n +1 ../zero-nine-three/go.mod ../one-zero-zero/go.mod
==> ../zero-nine-three/go.mod <==
module github.com/zero-nine-three

go 1.12

require github.com/prometheus/client_golang v0.9.3

==> ../one-zero-zero/go.mod <==
module github.com/one-zero-zero

go 1.12

require github.com/prometheus/client_golang v1.0.0

Then go mod tidy removes and go get adds the github.com/go-logfmt/logfmt v0.4.0 // indirect dependency:

$ go mod tidy

$ cat go.mod
module bug

go 1.12

replace github.com/one-zero-zero => ../one-zero-zero

replace github.com/zero-nine-three => ../zero-nine-three

require (
	github.com/one-zero-zero v0.0.0-00010101000000-000000000000
	github.com/zero-nine-three v0.0.0-00010101000000-000000000000
)

$ go get

$ cat go.mod
module bug

go 1.12

replace github.com/one-zero-zero => ../one-zero-zero

replace github.com/zero-nine-three => ../zero-nine-three

require (
	github.com/go-logfmt/logfmt v0.4.0 // indirect
	github.com/one-zero-zero v0.0.0-00010101000000-000000000000
	github.com/zero-nine-three v0.0.0-00010101000000-000000000000
)

We have noted that this behavior is not reproduced if we're importing a library without a hostname (i.e., zero-nine-three instead of github.com/zero-nine-three)

This behavior forces us to disable the tidyness check in our CI pipelines, since our tests are adding the indirect dependency.

@bcmills PTAL and ask for more info if needed.

P.D.: I've omitted the .go files in the example containing just underscore-imports to make tidy keep the imports, and the go.sum files.

bcmills · 2019-07-11T21:57:57Z

@colega, this was fixed as #26902 and will not be backported to the 1.12 branch. If you can still reproduce it (or something similar) using go1.13beta1, please open a new issue.

mvdan added the modules label Jan 12, 2019

mvdan changed the title ~~go mod tidy and go get -u fight eachother~~ cmd/go: mod tidy and get -u fight each other Jan 12, 2019

mvdan added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 12, 2019

bcmills added this to the Go1.13 milestone Jan 14, 2019

bcmills changed the title ~~cmd/go: mod tidy and get -u fight each other~~ cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u' Jan 14, 2019

bcmills mentioned this issue Jan 15, 2019

cmd/go: module update should drop unused indirect dependencies? #26474

Open

bcmills self-assigned this Feb 14, 2019

thepudds mentioned this issue Feb 28, 2019

cmd/go: go get -u in module mode can fail and produce output that's not easy to diagnose #30455

Closed

thepudds mentioned this issue Mar 14, 2019

cmd/go: "get -u" stumbles over repos imported via non-canonical paths #30831

Open

bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 6, 2019

gopherbot closed this as completed Jun 6, 2019

thepudds mentioned this issue Jul 14, 2019

cmd/go: go get -u on main module bloats go.mod with unnecessary dependencies #33102

Closed

zlozano mentioned this issue Aug 21, 2019

After build failed, I removed GOFLAGS='-mod=vendor', so that go get c… asecurityteam/asset-inventory-api#28

Merged

golang locked and limited conversation to collaborators Jul 10, 2020

gopherbot added the FrozenDueToAge label Jul 10, 2020

rsc unassigned bcmills Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u' #29702

cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u' #29702

SamWhited commented Jan 12, 2019 •

edited

mvdan commented Jan 12, 2019

bcmills commented Jan 14, 2019

bcmills commented Jan 14, 2019

bcmills commented Jan 14, 2019

SamWhited commented Jan 14, 2019 •

edited

bcmills commented Jan 14, 2019

SamWhited commented Jan 14, 2019

bcmills commented Jan 14, 2019

SamWhited commented Jan 14, 2019

rsc commented Jan 24, 2019

thanasik commented Apr 3, 2019 •

edited

bcmills commented Apr 9, 2019

thepudds commented May 5, 2019

gopherbot commented Jun 6, 2019

colega commented Jul 2, 2019

bcmills commented Jul 11, 2019

cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u' #29702

cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u' #29702

Comments

SamWhited commented Jan 12, 2019 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

mvdan commented Jan 12, 2019

bcmills commented Jan 14, 2019

bcmills commented Jan 14, 2019

bcmills commented Jan 14, 2019

SamWhited commented Jan 14, 2019 • edited

bcmills commented Jan 14, 2019

SamWhited commented Jan 14, 2019

bcmills commented Jan 14, 2019

SamWhited commented Jan 14, 2019

rsc commented Jan 24, 2019

thanasik commented Apr 3, 2019 • edited

bcmills commented Apr 9, 2019

thepudds commented May 5, 2019

gopherbot commented Jun 6, 2019

colega commented Jul 2, 2019

bcmills commented Jul 11, 2019

SamWhited commented Jan 12, 2019 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

SamWhited commented Jan 14, 2019 •

edited

thanasik commented Apr 3, 2019 •

edited