Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: 1.17 go mod tidy produces go.mod files that cannot be used with go mod graph to understand the source of dependencies #47648

Open
liggitt opened this issue Aug 11, 2021 · 6 comments
Labels
GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@liggitt
Copy link
Contributor

liggitt commented Aug 11, 2021

What version of Go are you using (go version)?

$ go version
go version go1.17rc2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/liggitt/Library/Caches/go-build"
GOENV="/Users/liggitt/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/liggitt/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/liggitt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/liggitt/.gvm/gos/go1.17rc2"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/liggitt/.gvm/gos/go1.17rc2/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17rc2"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/liggitt/go/src/k8s.io/kubernetes/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/7f/9xt_73f12xlby0w362rgk0s400kjgb/T/go-build1198370725=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

hoisted from discussion at #46366 (comment)

Tidied a module with direct and indirect dependencies and ran through dependency stats checking tools based on go mod graph.

go mod init example

echo '
package main

import (
  "fmt"
  "github.com/liggitt/tidy-117/b"
)

func main() {
  fmt.Println(b.Do())
}
' > main.go

go mod tidy

go mod graph

What did you expect to see?

Useful module graph output.

go1.16 produces this:

example github.com/liggitt/tidy-117/b@v1.0.0
github.com/liggitt/tidy-117/b@v1.0.0 github.com/liggitt/tidy-117/c@v1.0.0

What did you see instead?

go1.17rc2 produces this:

example github.com/liggitt/tidy-117/b@v1.0.0
example github.com/liggitt/tidy-117/c@v1.0.0
github.com/liggitt/tidy-117/b@v1.0.0 github.com/liggitt/tidy-117/c@v1.0.0

That shows an edge from example → github.com/liggitt/tidy-117/c which does not actually exist... it is only transitive via github.com/liggitt/tidy-117/b.

go mod graph is currently useful for understanding subtrees of tidied dependencies, and piping to graph visualization tools like dot to find candidates for reducing dependencies.

The 1.17 changes to go mod tidy add edges that make go mod graph unsuitable for this.

@liggitt
Copy link
Contributor Author

liggitt commented Aug 11, 2021

cc @bcmills @jayconrod @dims

@liggitt liggitt changed the title cmd/go: go mod tidy produces go.mod files that cannot be used with go mod graph to understand the source of dependencies cmd/go: 1.17 go mod tidy produces go.mod files that cannot be used with go mod graph to understand the source of dependencies Aug 11, 2021
@dmitshur dmitshur added GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Aug 11, 2021
@bcmills
Copy link
Contributor

bcmills commented Aug 11, 2021

Tidied a module with direct and indirect dependencies and ran through dependency stats checking tools based on go mod graph.

The 1.17 changes to go mod tidy add edges that make go mod graph unsuitable for this.

I don't think go mod graph was ever really suitable for computing direct vs. indirect dependency stats. Its main use-case is to help you figure out why a specific version is selected, not whether or how that version is relevant to the packages within your module.

The dependency edges added for go 1.17 modules could also be present in other modules any time a transitive dependency is upgraded above the version that would otherwise be selected. So the assumption that “explicit” == “direct” was already quite fragile (it did not necessarily hold, although it may have happened to hold for many projects in practice).

@bcmills
Copy link
Contributor

bcmills commented Aug 11, 2021

The tools that we have today for analyzing the reason a dependency is needed are go mod why (and go mod why -m) and go list.

There are also some third-party tools that build on top of those tools to provide more structure and support more detailed queries. (For example, some folks on the Gophers Slack this morning were recommending goda. I haven't used it myself but it seems to support a pretty reasonable query language.)

@liggitt
Copy link
Contributor Author

liggitt commented Aug 16, 2021

The dependency edges added for go 1.17 modules could also be present in other modules any time a transitive dependency is upgraded above the version that would otherwise be selected.

Understood, but in that scenario, the module had an explicit opinion about the transitive dep, so representing an edge made sense.

I think it's unfortunate that the graph representation we previously got for free using go mod tidy and go mod graph we now have to reconstruct from the output of go list

@liggitt
Copy link
Contributor Author

liggitt commented Sep 8, 2021

The tools that we have today for analyzing the reason a dependency is needed are go mod why (and go mod why -m) and go list.

Working on switching to use go list in kubernetes/kubernetes#104461 (comment), it looks like since go list is build-tag constrained, it doesn't include all of the packages/modules that inform go mod tidy and go mod vendor. Is that accurate? Is there a way to use go list to get the full set of dependencies from all build tags?

@bcmills
Copy link
Contributor

bcmills commented Sep 9, 2021

go list is indeed tag-constrained. (#42504 tracks the task of addressing that.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants