Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: 'go mod init' should validate module paths for v2+ dependencies #31549

Open
thepudds opened this issue Apr 18, 2019 · 3 comments
Open
Labels
GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@thepudds
Copy link
Contributor

thepudds commented Apr 18, 2019

What version of Go are you using (go version)?

go 1.12.4

Does this issue reproduce with the latest release?

Yes, including tip.

What did you do?

Observed problems in modules that were ultimately traced back to losing information in the go mod init process.

What did you expect to see?

go mod init setting proper module paths for v2+ dependencies when converting from a prior dependency manager.

What did you see instead?

go mod init never seems to set the /vN as part of the module path for a require statement

Example from scratch

This can happen in different ways in the wild, but here is an example from scratch.

$ export GO111MODULE=off
$ go get github.com/gobuffalo/packr/v2
$ mkdir -p $GOPATH/src/example.com/hello
$ cd $GOPATH/src/example.com/hello
$ cat <<EOF > hello.go
package hello
import _ "github.com/gobuffalo/packr/v2"
EOF

$ dep init
  ...
  Using ^2.2.0 as constraint for direct dep github.com/gobuffalo/packr
  ... 

$ export GO111MODULE=on
$ gotip mod init
go: creating new go.mod: module example.com/scratchpad/hello
go: copying requirements from Gopkg.lock

$ grep packr go.mod
        github.com/gobuffalo/packr v0.0.0-20190416161152-cda4ac255773

cda4ac255773 corresponds to the v2.2.0 tag, but the require in the created go.mod is missing the required /v2 after packr.

In other words, go mod init created an incorrect go.mod.

Gopkg.lock and Gopkg.toml created by this example
$ cat Gopkg.lock
# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.


[[projects]]
  digest = "1:d29ee5ef14a7e0253facd0bcebe6a69a7a4e02a67eb24d2aacd8ccb4a7cea6fc"
  name = "github.com/gobuffalo/envy"
  packages = ["."]
  pruneopts = "UT"
  revision = "043cb4b8af871b49563291e32c66bb84378a60ac"
  version = "v1.7.0"

[[projects]]
  branch = "master"
  digest = "1:5cc0801073158de62986111e35292e498e5d762a45da812bac744986fbe11daa"
  name = "github.com/gobuffalo/genny"
  packages = ["."]
  pruneopts = "UT"
  revision = "3ca520ef0d9ea4981534aae98bf9a003d9b18678"

[[projects]]
  branch = "master"
  digest = "1:7779ec172ea5898029020425bf58586c68c341fe4c3c23c2508605767de3ce47"
  name = "github.com/gobuffalo/gogen"
  packages = [
    ".",
    "goimports",
    "gomods",
  ]
  pruneopts = "UT"
  revision = "8f38393713f59beb43221f903c07f58f0377a063"

[[projects]]
  branch = "master"
  digest = "1:f391538f4166f7c0aa50943372af196304488b039fe663c134d42f064e1ed92c"
  name = "github.com/gobuffalo/logger"
  packages = ["."]
  pruneopts = "UT"
  revision = "86e12af44bc273e0063fad5b6ad588890b8cfe6b"

[[projects]]
  digest = "1:a3259d2a6c245b62e40232800e3ebb4d7d5840940638de0e4754cb13cdd37790"
  name = "github.com/gobuffalo/mapi"
  packages = ["."]
  pruneopts = "UT"
  revision = "0bb5e840be332d4280e40f2e6c50777c615bbac5"
  version = "v1.0.2"

[[projects]]
  branch = "master"
  digest = "1:8561bb77169b6b26a39bcf2945f582a10ccbe3533d644ada72094bd5fc86b4d5"
  name = "github.com/gobuffalo/packd"
  packages = ["."]
  pruneopts = "UT"
  revision = "a385830c7fc0495b810318788a043c7655da337d"

[[projects]]
  digest = "1:0fac441be66684180b031233313b8315644c0734fd73944c3e156e3cf5fbe203"
  name = "github.com/gobuffalo/packr"
  packages = [
    "v2",
    "v2/file",
    "v2/file/resolver",
    "v2/file/resolver/encoding/hex",
    "v2/jam/parser",
    "v2/plog",
  ]
  pruneopts = "UT"
  revision = "cda4ac25577350dac682c8715dc6659327232596"
  version = "v2.2.0"

[[projects]]
  branch = "master"
  digest = "1:b90ac64448d67ef218124185a8524f3c665fb7f51f096d5dd51ce950a1775d89"
  name = "github.com/gobuffalo/syncx"
  packages = ["."]
  pruneopts = "UT"
  revision = "33c29581e754bd354236e977dfe426e55331c45d"

[[projects]]
  digest = "1:ecd9aa82687cf31d1585d4ac61d0ba180e42e8a6182b85bd785fcca8dfeefc1b"
  name = "github.com/joho/godotenv"
  packages = ["."]
  pruneopts = "UT"
  revision = "23d116af351c84513e1946b527c88823e476be13"
  version = "v1.3.0"

[[projects]]
  digest = "1:38d2e838817528fd7d1ccdf7fa6700ee0f1727b3c484d13cab54bfbdaab12b0e"
  name = "github.com/karrick/godirwalk"
  packages = ["."]
  pruneopts = "UT"
  revision = "6d1c7760ec857d2984abe0a23fa263877f50d3f0"
  version = "v1.8.0"

[[projects]]
  digest = "1:31e761d97c76151dde79e9d28964a812c46efc5baee4085b86f68f0c654450de"
  name = "github.com/konsorten/go-windows-terminal-sequences"
  packages = ["."]
  pruneopts = "UT"
  revision = "f55edac94c9bbba5d6182a4be46d86a2c9b5b50e"
  version = "v1.0.2"

[[projects]]
  branch = "master"
  digest = "1:6e2ed1bdbf1d14b4d0be58bcd3f1c3000c1e226964354457b8e6ca69e83a1cbb"
  name = "github.com/markbates/oncer"
  packages = ["."]
  pruneopts = "UT"
  revision = "bf2de49a0be218916e69a11d22866e6cd0a560f2"

[[projects]]
  digest = "1:28687e854cec240942c103259668b132a8450b05a6cc677eea2282b26ae29310"
  name = "github.com/markbates/safe"
  packages = ["."]
  pruneopts = "UT"
  revision = "6fea05a5732486546a4836b7a1d596c5ec687b98"
  version = "v1.0.1"

[[projects]]
  digest = "1:cf31692c14422fa27c83a05292eb5cbe0fb2775972e8f1f8446a71549bd8980b"
  name = "github.com/pkg/errors"
  packages = ["."]
  pruneopts = "UT"
  revision = "ba968bfe8b2f7e042a574c888954fccecfa385b4"
  version = "v0.8.1"

[[projects]]
  digest = "1:e09ada96a5a41deda4748b1659cc8953961799e798aea557257b56baee4ecaf3"
  name = "github.com/rogpeppe/go-internal"
  packages = [
    "modfile",
    "module",
    "semver",
  ]
  pruneopts = "UT"
  revision = "438578804ca6f31be148c27683afc419ce47c06e"
  version = "v1.3.0"

[[projects]]
  digest = "1:fd61cf4ae1953d55df708acb6b91492d538f49c305b364a014049914495db426"
  name = "github.com/sirupsen/logrus"
  packages = ["."]
  pruneopts = "UT"
  revision = "8bdbc7bcc01dcbb8ec23dc8a28e332258d25251f"
  version = "v1.4.1"

[[projects]]
  branch = "master"
  digest = "1:bbe51412d9915d64ffaa96b51d409e070665efc5194fcf145c4a27d4133107a4"
  name = "golang.org/x/crypto"
  packages = ["ssh/terminal"]
  pruneopts = "UT"
  revision = "b43e412143f90fca62516c457cae5a8dc1595586"

[[projects]]
  branch = "master"
  digest = "1:e18a5437c48ea1bd6dcd960efdd634d05ae5c2dd033d2d17f73cc20be92895ac"
  name = "golang.org/x/sys"
  packages = [
    "unix",
    "windows",
  ]
  pruneopts = "UT"
  revision = "f0ce4c0180bef7e9c51babed693a6e47fdd8962f"

[[projects]]
  branch = "master"
  digest = "1:be1ab6d2b333b1d487c01f1328aef9dc76cee4ff4f780775a552d2a1653f0207"
  name = "golang.org/x/tools"
  packages = [
    "go/ast/astutil",
    "go/gcexportdata",
    "go/internal/gcimporter",
    "go/internal/packagesdriver",
    "go/packages",
    "go/types/typeutil",
    "imports",
    "internal/fastwalk",
    "internal/gopathwalk",
    "internal/module",
    "internal/semver",
  ]
  pruneopts = "UT"
  revision = "6732636ccdfd99c4301d1d1ac2307f091331f767"

[solve-meta]
  analyzer-name = "dep"
  analyzer-version = 1
  input-imports = ["github.com/gobuffalo/packr/v2"]
  solver-name = "gps-cdcl"
  solver-version = 1

$ cat Gopkg.toml

# Gopkg.toml example
#
# Refer to https://golang.github.io/dep/docs/Gopkg.toml.html
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
#   name = "github.com/user/project"
#   version = "1.0.0"
#
# [[constraint]]
#   name = "github.com/user/project2"
#   branch = "dev"
#   source = "github.com/myfork/project2"
#
# [[override]]
#   name = "github.com/x/y"
#   version = "2.4.0"
#
# [prune]
#   non-go = false
#   go-tests = true
#   unused-packages = true


[[constraint]]
  name = "github.com/gobuffalo/packr"
  version = "2.2.0"

[prune]
  go-tests = true
  unused-packages = true

Additional comments

This issue is not about how to resolve this manually, which I know how to do. Rather, it would be better to not to have to resolve manually, but more importantly would better if others did not encounter follow-on issues due to this behavior. Also, in some cases go mod tidy or similar can let someone be "lucky" and not notice this, but in some cases the version information converted by go mod init is important to avoid follow-on issues.

This was a less significant problem early in the history of modules because not many v2+ packages had adopted modules yet.

However, more v2+ dependencies are adopting go.mod files themselves as modules move towards being on by default in 1.13.

I have seen this behavior be the root cause of hard-to-diagnose problems in the wild, and the rate of those problems likely would increase with more v2+ modules in the ecosystem. (And then later, this problem rate of course would decrease, given it is transitional in nature).

This is related to #30161, but that suggested an alternative solution of dropping the dependency during the conversion process, and that issue was ultimately closed by the reporter after the current behavior was explained and the reporter understood how to manually work around. However, ideally go mod init would record the right information here without the need for someone to read documentation or manually work around.

@dmitshur dmitshur added GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Apr 18, 2019
@dmitshur dmitshur added this to the Go1.13 milestone Apr 18, 2019
@bcmills bcmills changed the title cmd/go: 'go mod init' should record correct module paths for v2+ dependencies cmd/go: 'go mod init' should validate module paths for v2+ dependencies Apr 19, 2019
@bcmills
Copy link
Contributor

bcmills commented Apr 19, 2019

Somewhat related to #29814.

@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@dmitshur
Copy link
Contributor

dmitshur commented Mar 3, 2020

I know there have been some improvements in go mod init in this area, so this may be in scope of that.

Additionally, this may be related to (or already implemented in) the experimental gorelease tool. /cc @jayconrod

/cc @jayconrod @matloob @bcmills per owners.

@thepudds
Copy link
Contributor Author

thepudds commented Sep 29, 2020

Hi @bcmills, as far as I could tell from a quick look, go mod init in go 1.15.3 seems to be falsely complaining more than it should about finding /v2, /v5, and /v22 in import paths in vendor.conf. A possible example of this is in moby/moby#41504 (comment).

This is probably the wrong place to make this comment, but here we are 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants