Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go mod tidy updates a dependency unnecessarily #33771

Closed
leighmcculloch opened this issue Aug 21, 2019 · 12 comments
Closed

cmd/go: go mod tidy updates a dependency unnecessarily #33771

leighmcculloch opened this issue Aug 21, 2019 · 12 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@leighmcculloch
Copy link
Contributor

leighmcculloch commented Aug 21, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13rc1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN="/home/leighmcculloch/local/bin"
GOCACHE="/home/leighmcculloch/.cache/go-build"
GOENV="/home/leighmcculloch/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/leighmcculloch/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/leighmcculloch/local/bin/go/1.13rc1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/leighmcculloch/local/bin/go/1.13rc1/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/leighmcculloch/devel/stellar-go/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build871348672=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I initialized a new Go module in a repository that had a dep Gopkg.toml and Gopkg.lock while attempting to upgrade a repo from dep to modules (stellar/go#1634).

git clone https://github.com/leighmcculloch/stellar-go
cd stellar-go
git checkout gomod
git checkout 13ffc7fc2f5871f0a3a2d3b1878ddf5d53d80acf
go mod init github.com/stellar/go
git add .
go mod tidy
git diff

What did you expect to see?

I expected to see no changes to go.mod between a go mod init and a go mod tidy, because the file should be tidy already, as it was just created.

What did you see instead?

The go.mod file contains in 122 changes, many involving appending // indirect. The change I'm particularly concerned with is where a dependency is being updated unnecessarily.

After the go mod init, this dependency was included, which is expected:

github.com/BurntSushi/toml v0.2.1-0.20160717150709-99064174e013

After the go mod tidy, that dependency was changed to its latest tagged version:

github.com/BurntSushi/toml v0.3.1

The go mod tidy added dependencies from test files, which were pruned in the original Gopkg.* files, so I understand there should be some changes and additions. But when I ask go mod why about the dependency, the only package using the dependency is not a test dependency:

# github.com/BurntSushi/toml
github.com/stellar/go/clients/stellartoml
github.com/BurntSushi/toml

I don't see a reason why the go mod tidy would have updated the dependency since there are no new test packages dependent on it that might have had a higher required version defined.

Interestingly if I change the version back and run go mod tidy a second time, the file stays unchanged. I must have made a mistake, as when I run it a second time now, it changes it.

@leighmcculloch
Copy link
Contributor Author

In my statements above I misunderstood the behavior of go mod why. I thought it would show me all the reasons why a module was being included at a specific version.

Using the command go mod graph command instead I can see this other dependency:

github.com/spf13/cobra@v0.0.5 github.com/BurntSushi/toml@v0.3.1

It states there's a dependency on v0.3.1, but cobra at v0.0.5 did not have a go.mod file, or any dependency file indicating a v0.3.1+ was required. The cobra module was also updated during the go mod tidy, as it was pinned at an earlier revision before that. It was originally:

github.com/spf13/cobra v0.0.0-20160830174925-9c28e4bbd74e

@bcmills
Copy link
Contributor

bcmills commented Aug 22, 2019

It states there's a dependency on v0.3.1, but cobra at v0.0.5 did not have a go.mod file, or any dependency file indicating a v0.3.1+ was required.

Um... I think you're mistaken?
https://github.com/spf13/cobra/blob/f2b07da1e2c38d5f12845a4f607e2e1018cbb1f5/go.mod#L6

The cobra module was also updated during the go mod tidy, as it was pinned at an earlier revision before that.

The go.mod file does not “pin” dependencies: it only sets minimums, which can be raised by minimums in other transitive dependencies. What does go mod graph tell you about
transitive requirements on github.com/spf13/cobra v0.0.5?

@bcmills
Copy link
Contributor

bcmills commented Aug 22, 2019

In my statements above I misunderstood the behavior of go mod why. I thought it would show me all the reasons why a module was being included at a specific version.

We certainly do need better diagnostics for “why is this module at this version?”.

That's #33124. (CC @jayconrod @jadekler)

@bcmills
Copy link
Contributor

bcmills commented Aug 22, 2019

As far as I can tell this is working as designed: go mod init should have generated the closest approximation to your Gopkg.lock that it could, and then go mod tidy should have made the configuration consistent with the requirements listed in your transitive dependencies.

Please investigate using go mod graph and let us know if you spot any more unexplained upgrades.

@bcmills bcmills added modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Aug 22, 2019
@bcmills bcmills added this to the Go1.14 milestone Aug 22, 2019
@leighmcculloch
Copy link
Contributor Author

leighmcculloch commented Aug 22, 2019

@bcmills Thanks for responding so fast! Thanks for pointing out my mistake. The issue isn't with resolving the dependency for cobra, because you're correct that cobra v0.0.5 does depend on a min of v0.3.1.

I need to dig deeper because before running go mod tidy cobra was set to a min of v0.0.0-20160830174925-9c28e4bbd74e. There must be some series of dependencies in here that are causing it to bump upwards even though in the Go Dep files it was quite happy at that revision.

I'll investigate further and come back to here when I know more.

@leighmcculloch
Copy link
Contributor Author

leighmcculloch commented Aug 28, 2019

I managed to through a series of go get commands downgrade things to get the minimum versions set back to what they should have been in the situation above.


I've discovered a new situation though in the same repository where go mod tidy repeatedly increases the minimum version, and go mod why, go mod graph, and my own inspection of the dependencies don't point to a reason for the minimum to be higher. My main module is the only module that has go-metrics as a dependency but go mod tidy keeps bumping my minimum version to be the latest commit on the repository.

To reproduce:

$ git clone git@github.com:stellar/go

$ cd go

$ git checkout e39f70cea9dd9e9fd695fbfedd43457c5a630f53

$ go mod init github.com/stellar/go

$ go mod tidy

$ cat go.mod | grep github.com/rcrowley/go-metrics 
        github.com/rcrowley/go-metrics v0.0.0-20190826022208-cac0b30c2563

$ go get github.com/rcrowley/go-metrics@a5cfc242a56b
go: finding github.com/rcrowley/go-metrics a5cfc242a56b

$ cat go.mod | grep github.com/rcrowley/go-metrics  
        github.com/rcrowley/go-metrics v0.0.0-20150601143443-a5cfc242a56b

$ go mod tidy

$ cat go.mod | grep github.com/rcrowley/go-metrics
        github.com/rcrowley/go-metrics v0.0.0-20190826022208-cac0b30c2563

$ go mod why -m github.com/rcrowley/go-metrics
# github.com/rcrowley/go-metrics
github.com/stellar/go/services/horizon/internal
github.com/rcrowley/go-metrics

$ go mod graph | grep github.com/rcrowley/go-metrics 
github.com/stellar/go github.com/rcrowley/go-metrics@v0.0.0-20190826022208-cac0b30c2563

This looks like a bug to me.

@leighmcculloch
Copy link
Contributor Author

leighmcculloch commented Aug 28, 2019

Doh, I posted that after hours of 👀 and figured out what's happening a few minutes after posting. But I only figured it out after attempting the process in go1.12.9, which had different logs and those logs pointed to me what was happening.

In the go mod why, go mod graph and go get output the only repo paths relating to the go-metrics module was the root module package. But in fact there was a dependency from one of my dependencies tests through to the go-metrics/exp package which is only available in a newer version.

When I ran the go mod tidy command with go1.13rc1 I saw no output:

$ go get github.com/rcrowley/go-metrics@a5cfc242a56b
go: finding github.com/rcrowley/go-metrics a5cfc242a56b
$ go mod tidy

When I ran the go mod tidy command with go1.12.9 I saw:

$ go1.12.9 get github.com/rcrowley/go-metrics@a5cfc242a56b
$ go1.12.9 mod tidy
go: finding github.com/rcrowley/go-metrics/exp latest
go: finding github.com/rcrowley/go-metrics latest

That log line was the clue to point to where the dependency was coming from.

So I think this issue can be closed, because the tidy command is upgrading the minimums to what the minimum actually is based on the dependency graph's full state, but the go mod why and go mod graph aren't displaying the entire dependency tree. The graph command seems like it would aid in understanding a lot, but it isn't displaying test dependencies.

@bcmills
Copy link
Contributor

bcmills commented Aug 28, 2019

When I ran the go mod tidy command with go1.13rc1 I saw no output:

#33284 should help with that.

go mod why and go mod graph aren't displaying the entire dependency tree

go mod graph tells you about module-level dependencies (via explicit entries in go.mod files). It doesn't distinguish between test and non-test dependencies: they should all end up in the module graph either way.

go mod why should definitely include test dependencies — please file steps to reproduce (separately) if it isn't.

Note that go mod why gives a path to any package that uses a module, not necessarily the one that imposes the selected version (that's #33124).

go mod why -m also stops at the end of the import graph — it doesn't complete the paths using module-level dependencies from the module graph (that's #27900).

@bcmills
Copy link
Contributor

bcmills commented Aug 28, 2019

It sounds like go mod tidy is working as designed, so I'm going to close this issue.

We certainly do appreciate the feedback about sources of confusion, though — hopefully fixes for the other issues linked above will help with that.

@bcmills bcmills closed this as completed Aug 28, 2019
@leighmcculloch
Copy link
Contributor Author

leighmcculloch commented Aug 30, 2019

I've got another case where go mod tidy is forcibly updating a dependency. I've read the linked issues but it's not clear to me if they will indicate why this upgrade is occurring. I've used go list to investigate and I see no relationship that would describe this upgrade.

To reproduce:

$ git clone https://github.com/leighmcculloch/stellar-go
$ git checkout gomod
$ git checkout b7c4a7060afe02e802a43fa7a06b716d26bd349d

$ go get github.com/rcrowley/go-metrics@a5cfc242a56b
go: finding github.com/rcrowley/go-metrics a5cfc242a56b

$ grep github.com/rcrowley/go-metrics -- go.mod
        github.com/rcrowley/go-metrics v0.0.0-20150601143443-a5cfc242a56b

$ go mod tidy
go: finding github.com/rcrowley/go-metrics latest

$ grep github.com/rcrowley/go-metrics -- go.mod
        github.com/rcrowley/go-metrics v0.0.0-20190826022208-cac0b30c2563

My assumption was that there must be a dependency that needs this version. go mod why did show me one dependency, but I looked at it, and there's no go.mod in that repo. This was the go mod why output:

$ go mod why github.com/rcrowley/go-metrics/...
# github.com/rcrowley/go-metrics
github.com/stellar/go/services/horizon/internal
github.com/rcrowley/go-metrics

# github.com/rcrowley/go-metrics/cmd/metrics-bench
(main module does not need package github.com/rcrowley/go-metrics/cmd/metrics-bench)

# github.com/rcrowley/go-metrics/cmd/metrics-example
(main module does not need package github.com/rcrowley/go-metrics/cmd/metrics-example)

# github.com/rcrowley/go-metrics/cmd/never-read
(main module does not need package github.com/rcrowley/go-metrics/cmd/never-read)

# github.com/rcrowley/go-metrics/exp
github.com/stellar/go/services/bifrost/ethereum
github.com/ethereum/go-ethereum/core/types
github.com/ethereum/go-ethereum/trie
github.com/ethereum/go-ethereum/trie.test
github.com/ethereum/go-ethereum/ethdb
github.com/ethereum/go-ethereum/metrics
github.com/rcrowley/go-metrics/exp

# github.com/rcrowley/go-metrics/librato
(main module does not need package github.com/rcrowley/go-metrics/librato)

# github.com/rcrowley/go-metrics/stathat
(main module does not need package github.com/rcrowley/go-metrics/stathat)

I figured there must be another dependency, so instead I used this go list command to find every package and check all it's imports to find what other module must have code importing the package.

$ go list -f '{{.ImportPath}}:{{$.Imports}}' all | grep 'github.com/rcrowley/go-metrics' | cut -d':' -f1
github.com/ethereum/go-ethereum/ethdb
github.com/ethereum/go-ethereum/metrics
github.com/ethereum/go-ethereum/trie
github.com/rcrowley/go-metrics
github.com/rcrowley/go-metrics/exp
github.com/stellar/go/services/horizon/internal
github.com/stellar/go/services/horizon/internal/ingest
github.com/stellar/go/services/horizon/internal/logmetrics
github.com/stellar/go/services/horizon/internal/txsub

From best I can tell the only packages dependent on github.com/rcrowley/go-metrics are:

  • github.com/stellar/go (the main module)
  • github.com/ethereum/go-ethereum

The only dependency doesn't have a go.mod file and it appears like a bug that go mod tidy is forcing the version to latest.

I've iterated through all the commits in the github.com/rcrowley/go-metrics and attempted to go get ... && go mod tidy every commit, and this commit is the oldest commit that go mod tidy won't force to latest: rcrowley/go-metrics@51425a2

The commit before that commit when go get ... && go mod tidy causes go.mod to be updated to it's latest revision: rcrowley/go-metrics@8ffdf55

Am I missing something? It seems like go mod has no reason to force github.com/rcrowley/go-metrics to its latest revision.

@leighmcculloch
Copy link
Contributor Author

I realize the go list command I included above didn't look for usage in tests, xtests, or for all dependencies, but when I add those things it doesn't show up any new packages:

$ go list -deps -test -f '{{.ImportPath}}:{{.Imports}}:{{.TestImports}}:{{.XTestImports}}' all | grep 'github.com/rcrowley/go-metrics' | cut -d':' -f1                 
github.com/rcrowley/go-metrics
github.com/ethereum/go-ethereum/trie
github.com/rcrowley/go-metrics/exp
github.com/ethereum/go-ethereum/metrics
github.com/ethereum/go-ethereum/ethdb
github.com/stellar/go/services/horizon/internal/txsub
github.com/stellar/go/services/horizon/internal/ingest
github.com/stellar/go/services/horizon/internal/logmetrics
github.com/stellar/go/services/horizon/internal
github.com/ethereum/go-ethereum/trie [github.com/ethereum/go-ethereum/trie.test]
github.com/rcrowley/go-metrics [github.com/rcrowley/go-metrics.test]
github.com/rcrowley/go-metrics.test
github.com/stellar/go/services/horizon/internal [github.com/stellar/go/services/horizon/internal.test]
github.com/stellar/go/services/horizon/internal/ingest [github.com/stellar/go/services/horizon/internal/ingest.test]
github.com/stellar/go/services/horizon/internal/logmetrics [github.com/stellar/go/services/horizon/internal/logmetrics.test]
github.com/stellar/go/services/horizon/internal/txsub [github.com/stellar/go/services/horizon/internal/txsub.test]

@leighmcculloch
Copy link
Contributor Author

I've discovered why this is happening, and I suspect it will be difficult for go mod why or go mod graph to communicate this.

Via github.com/ethereum/go-ethereum/trie.test I am dependent on github.com/rcrowley/go-metrics/exp. The exp sub-package was added in 51425a2.

go mod tidy must be seeing my dependencies test being dependent on the exp subpackage, seeing that it is not present in a5cfc242a56b and pulling latest to get it.

leighmcculloch added a commit to stellar/go that referenced this issue Aug 30, 2019
…d2 (#1679)

Update `github.com/rcrowley/go-metrics` for transition to Modules (#1634).

From: rcrowley/go-metrics@a5cfc242a56b
To: rcrowley/go-metrics@51425a2415d2

Diff: rcrowley/go-metrics@a5cfc24...51425a2 (41 commits)

To make the transition to Modules possible. Some dependencies at their current version have incompatible relationships with other dependencies we have. Dep was more tolerant to incompatible versions than Modules. Making this change ahead of the PR that switches us to Modules makes that PR a functional no-op.

The reason Modules is updating `go-metrics` is because the monorepo is also dependent on `github.com/ethereum/go-ethereum` which is in turn, via it's tests, dependent on the `exp` sub-package within `go-metrics`. The sub-package was added in rcrowley/go-metrics@51425a2.

The `go` tool in this situation will automatically upgrade `go-metrics` to it's latest revision since it can see the `exp` is available there. This PR is however upgrading to the first or oldest revision that contains the sub-package we need to satisfy Modules. We could upgrade to the latest revision but the changes in the smaller diff are much less significant and pose a smaller risk than the changes that are in the diff (rcrowley/go-metrics@a5cfc24...cac0b30 144 commits) to take us all the way to latest. 

More details about how this was discovered can be found here: golang/go#33771 (comment)
@golang golang locked and limited conversation to collaborators Aug 29, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

3 participants