Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go get $module@$commit should resolve it to the highest appropriate pseudo-version #27171

Closed
rogpeppe opened this issue Aug 23, 2018 · 35 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@rogpeppe
Copy link
Contributor

rogpeppe commented Aug 23, 2018

go version go1.11rc2 linux/amd64

With my existing $GOPATH, I can run go get to get a particular commit, and it doesn't use the correct version as I'd expect from https://go-review.googlesource.com/c/go/+/124515, but downgrades instead. With a fresh $GOPATH, it works OK, so I suspect that somehow the cached info isn't being updated appropriately.

Here's what I did:

% cd /tmp
% mkdir modtest
% cd modtest
% go mod init example.com/foo/bar
go: creating new go.mod: module example.com/foo/bar
% cat > main.go
package main
import _ "github.com/frankban/quicktest"

func main() {
}
% go mod tidy
% go list -m all | grep cmp
github.com/google/go-cmp v0.2.0
% go get -x github.com/google/go-cmp@5411ab924f9ffa6566244a9e504bc347edacffd3
# /home/rog/src/go/pkg/mod/cache/vcs/bf8dbe084a71342417894c38c6144465bdc383bd4eb51954ad4212efcf8a7445 for git2 https://github.com/frankban/quicktest
cd /home/rog/src/go/pkg/mod/cache/vcs/bf8dbe084a71342417894c38c6144465bdc383bd4eb51954ad4212efcf8a7445; git ls-remote -q https://github.com/frankban/quicktest
1.303s # cd /home/rog/src/go/pkg/mod/cache/vcs/bf8dbe084a71342417894c38c6144465bdc383bd4eb51954ad4212efcf8a7445; git ls-remote -q https://github.com/frankban/quicktest
# /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e for git2 https://github.com/google/go-cmp
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
1.243s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
% cat go.mod
module example.com/foo/bar

require (
	github.com/frankban/quicktest v0.8.0
	github.com/google/go-cmp v0.0.0-20180328201512-5411ab924f9f // indirect
	github.com/kr/pretty v0.1.0 // indirect
)
% 

Note that despite getting the latest master commit of go-cmp, it has downgraded the quicktest package inappropriately, and at no point did it run the git describe command which would have told it which version to use. If I use @master, it works correctly:

% GOPATH=$h/src/go
% echo module example.com/foo/bar > go.mod
% go mod tidy
% cat go.mod
module example.com/foo/bar

require github.com/frankban/quicktest v1.1.0
% go get -x github.com/google/go-cmp@master
# /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e for git2 https://github.com/google/go-cmp
go: finding github.com/google/go-cmp master
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git tag -l
0.004s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git tag -l
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
1.300s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
0.004s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
0.005s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
0.007s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
0.007s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
% cat go.mod
module example.com/foo/bar

require (
	github.com/frankban/quicktest v1.1.0
	github.com/google/go-cmp v0.2.1-0.20180328201512-5411ab924f9f // indirect
)

If I use a fresh $GOPATH, it also works OK:

% echo module example.com/foo/bar > go.mod
% go mod tidy
go: finding github.com/frankban/quicktest v1.1.0
go: downloading github.com/frankban/quicktest v1.1.0
go: finding github.com/google/go-cmp v0.2.0
go: finding github.com/kr/pretty v0.1.0
go: finding github.com/kr/text v0.1.0
go: finding github.com/kr/pty v1.1.1
go: downloading github.com/google/go-cmp v0.2.0
go: downloading github.com/kr/pretty v0.1.0
go: downloading github.com/kr/text v0.1.0
% cat go.mod
module example.com/foo/bar

require github.com/frankban/quicktest v1.1.0
% go list -m all | grep cmp
github.com/google/go-cmp v0.2.0
% go get -x github.com/google/go-cmp@5411ab924f9ffa6566244a9e504bc347edacffd3
# /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e for git2 https://github.com/google/go-cmp
go: finding github.com/google/go-cmp 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
0.003s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git tag -l
0.002s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git tag -l
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
1.328s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git fetch -f --depth=1 https://github.com/google/go-cmp 5411ab924f9ffa6566244a9e504bc347edacffd3:refs/dummy
1.477s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git fetch -f --depth=1 https://github.com/google/go-cmp 5411ab924f9ffa6566244a9e504bc347edacffd3:refs/dummy
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
0.002s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
0.002s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
0.002s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c protocol.version=0 fetch --unshallow -f https://github.com/google/go-cmp 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
1.729s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c protocol.version=0 fetch --unshallow -f https://github.com/google/go-cmp 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
0.014s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9f
0.006s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9f
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
0.004s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
0.003s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
go: downloading github.com/google/go-cmp v0.2.1-0.20180328201512-5411ab924f9f
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
0.002s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 5411ab924f9ffa6566244a9e504bc347edacffd3
0.018s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 5411ab924f9ffa6566244a9e504bc347edacffd3
% 
% cat go.mod
module example.com/foo/bar

require (
	github.com/frankban/quicktest v1.1.0
	github.com/google/go-cmp v0.2.1-0.20180328201512-5411ab924f9f // indirect
)
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 16, 2018
@bcmills bcmills added this to the Go1.12 milestone Nov 16, 2018
@bcmills
Copy link
Contributor

bcmills commented Nov 16, 2018

github.com/google/go-cmp v0.0.0-20180328201512-5411ab924f9f // indirect

The v0.0.0 pseudo-version is the problem: go get sees that and decides that it needs to downgrade quicktest, because the previous quicktest version required v0.2.0.

I don't know how you got an entry mapping that commit to v0.0.0 in your module cache, given that it's after the v0.2.0 tag, but I'm not surprised: we are a bit sloppy about the pseudo-versions we allow (#27173). Perhaps you resolved it as v0.0.0 via an older go.mod file at some point?

At any rate, if you go get the go-cmp repo with a higher-numbered pseudoversion, you'll find that it sticks:

go: creating new go.mod: module golang.org/issue/27171

$ go get -m github.com/google/go-cmp@v0.0.0-20180328201512-5411ab924f9f
go: finding github.com/google/go-cmp v0.0.0-20180328201512-5411ab924f9f

$ go get -m github.com/frankban/quicktest@v1.1.0
go: finding github.com/frankban/quicktest v1.1.0
go: finding github.com/google/go-cmp v0.2.0
go: finding github.com/kr/pretty v0.1.0
go: finding github.com/kr/text v0.1.0
go: finding github.com/kr/pty v1.1.1

$ go list -m all
golang.org/issue/27171
github.com/frankban/quicktest v1.1.0
github.com/google/go-cmp v0.2.0
github.com/kr/pretty v0.1.0
github.com/kr/pty v1.1.1
github.com/kr/text v0.1.0

$ go get -m github.com/google/go-cmp@5411ab924f9f
go: finding github.com/frankban/quicktest v1.0.0
go: finding github.com/frankban/quicktest v0.9.1
go: finding github.com/frankban/quicktest v0.9.0
go: finding github.com/frankban/quicktest v0.8.0

$ go list -m all
golang.org/issue/27171
github.com/frankban/quicktest v0.8.0
github.com/google/go-cmp v0.0.0-20180328201512-5411ab924f9f
github.com/kr/pretty v0.1.0
github.com/kr/pty v1.1.1
github.com/kr/text v0.1.0

$ go get -m github.com/google/go-cmp@v0.2.1-0.20180328201512-5411ab924f9f
go: finding github.com/google/go-cmp v0.2.1-0.20180328201512-5411ab924f9f

$ go get -m github.com/frankban/quicktest@v1.1.0

$ go get -m github.com/google/go-cmp@5411ab924f9f

$ go list -m all
golang.org/issue/27171
github.com/frankban/quicktest v1.1.0
github.com/google/go-cmp v0.2.1-0.20180328201512-5411ab924f9f
github.com/kr/pretty v0.1.0
github.com/kr/pty v1.1.1
github.com/kr/text v0.1.0

@bcmills bcmills changed the title cmd/go: go get $module@$commit sometimes downgrades inappropriately cmd/go: go get $module@$commit should resolve it to the highest appropriate pseudo-version Nov 16, 2018
@bcmills bcmills modified the milestones: Go1.12, Go1.13 Nov 16, 2018
@hyangah
Copy link
Contributor

hyangah commented Jan 16, 2019

Another observation.
Seems like the pseudo-version generation is sensitive to what's in the module cache.

$ export GOPATH=$(mktemp -d)
$ go list -m -json golang.org/x/text@master
$ cat go.mod
module foo

go 1.12

$ go list -m -json golang.org/x/text@master
go: finding golang.org/x/text master
{
	"Path": "golang.org/x/text",
	"Version": "v0.0.0-20181227161524-e6919f6577db",
	"Time": "2018-12-27T16:15:24Z"
}

$ go list -m -json golang.org/x/text@master
go: finding golang.org/x/text master
{
	"Path": "golang.org/x/text",
	"Version": "v0.3.1-0.20181227161524-e6919f6577db",
	"Time": "2018-12-27T16:15:24Z"
}

Note that two subsequent go list commands (back to back) resulted in two different pseudo versions based on the same hash. It seems the pseudo-version generation is sensitive to what is in the module cache.

$ go version
go version devel +5fae09b738 Tue Jan 15 23:30:58 2019 +0000 darwin/amd64

@thepudds
Copy link
Contributor

thepudds commented Jan 17, 2019

Working with docker with Go modules is currently a bit convoluted for multiple reasons (e.g., sirupsen/logrus#799 (comment)). There are solutions (e.g., see #28489 (comment), #28489 (comment), and others), but this issue here adds to the complexity.

For example, you can have something like the following version (from early 2018) in one go.mod:

$ go1.12beta1 get github.com/docker/docker@0ede01237c9a
$ grep github.com/docker/docker go.mod

require github.com/docker/docker v0.0.0-20180221164450-0ede01237c9a // indirect

but then because that pseudo-version seems to incorrectly use v0.0.0, it is treated as happening prior to the latest syntactically correct semver version tag for docker of v1.13.1 (from early 2017), and you get thrown to the older v1.13.1 as the selected version in your build rather than the newer commit you specified, which is not the correct end-to-end result. (The semver comparison itself is correct, but the pseudo-version fed into the comparison is incorrect, which I think is this issue here, if following).

On top of that, v1.13.1 shows up for docker probably more than it should, either due to the "expected" reasons related to having dependencies that do not themselves have a go.mod, or in some cases I suspect perhaps due to other module issues.

CC @vito-c

@thepudds
Copy link
Contributor

thepudds commented Mar 6, 2019

This write-up here is based on a older quick look, so please take with grain of salt.

Depending on how you count, there might be 2-3 different things going on here:

  1. Docker and other repos that have tags that match a glob pattern of v[0-9]*.[0-9]*.[0-9]* can end up with v0.0.0 pseudo-versions instead of finding a proper semver tag to use as input to building a pseudo-version (when then causes problems when the v0.0.0 pseudo-version is fed into MVS). There is a RecentTag method that is defined to be "best effort", but not 100% clear if the caller handles the cases when the result is incorrect. describe here in RecentTag returns success here even if the result will be effectively rejected by the caller of RecentTag when the caller then calls tagToVersion. Perhaps that is deliberate, or perhaps that was just the first cut implementation.
	// RecentTag returns the most recent tag at or before the given rev
	// with the given prefix. It should make a best-effort attempt to
	// find a tag that is a valid semantic version (following the prefix),
	// or else the result is not useful to the caller, but it need not
	// incur great expense in doing so. 

This issue crops up in multiple repos, including notably docker, but a simplified constructed example showing a bad result vs. a good result:

# incorrectly records v0.0.0-20181229213331-5f2c3dc5628b, seemingly due to tag v18.06.16-ce
$ go get github.com/thepudds/nomodlib@5f2c3dc5628b

# correctly records v1.0.1-0.20181229213050-3e4103ddf9e0
$ go get github.com/thepudds/nomodlib@3e4103ddf9e0
  1. In cmd/go: pseudo-versions for post-v1 commits use the wrong major version #29262, rsc.io/quote at master has two different major versions following the Major Subdirectory approach. It has a v1 module in the root of the repo, and a v3 module in a /v3 subdirectory. If you ask for go get rsc.io/quote@master, that is asking for v1 of rsc.io/quote, which is a valid request, but the most recent tag on master matching that glob pattern ends up being v3.0.0, which I think is what is returned by RecentTag. The v3 is not what is desired, and the caller gives up and falls back to a v0.0.0- pseudo version (probably here). This could be viewed as a variation of 1., or could be viewed as a separate problem.

  2. golang/xtext in cmd/go: go get $module@$commit should resolve it to the highest appropriate pseudo-version #27171 (comment) seems to get different results for the tag returned by git describe on later runs of go get. Shot in the dark -- perhaps more git history is fetched as part of one of the earlier go get invocations, but that happens too late in that earlier go get to find the v0.3.x tag, but that additional history in the cache then makes a later go get find the v0.3.x tag?

Separately, I have not looked at the original report in #27171 (comment). Perhaps that falls into one of these three buckets, or perhaps it was just an older go.mod from vgo, or perhaps that is a distinct issue.

@leitzler
Copy link
Contributor

I'm trying to wrap my head around pseudo-versions and how they are intended to work, in particular the vn.n.n-0.date-hash form. The current documentation in pseudo.go states that:

If the most recent tagged version before the target commit is vX.Y.Z or vX.Y.Z+incompatible, then the pseudo-version uses form (2) or (3), [...]

What is the exact definition of "most recent" here? Is it the most recent in the current branch, the most recent that is merged into the current branch or the most recent that exists on the current branch? (or something else).

As an example, if I have this tree:

*   5aaa858 (HEAD -> master) Merge branch 'branch2'
|\  
| * 56bdc7d (tag: v0.2.2, branch2) ...
| * 1ce5c87 ...
* | 8d46cd4 (tag: v0.2.1) .
|/  
* 987d3d9 (tag: v0.2.0) .
*   ba5332a Merge branch 'branch1'
|\  
| * 620c636 (branch1) ..
| * 64deab7 (tag: v0.1.2) ..
| * 12d424d (tag: v0.1.1) ..
| * 070f504 .
* | 6c864f2 .
* | 3e081ae .
* | 98b862a (tag: v0.1.3) .
|/  
* b8cb65c (tag: v0.1.0) .
* 8b8037c .
* c16c020 (origin/master) empty readme

.. and runs go get <repo>@master should I expect a pseudo version based on:

  • v0.2.2 - the most recent in creation date, that is merged into master
  • v0.2.1 - the most recent, that is tagged on master
    (or something else?)

@bcmills
Copy link
Contributor

bcmills commented Apr 24, 2019

What is the exact definition of "most recent" here?

It is intended to be the semantically-latest tag that appears on any commit that is a (transitive) parent of the commit with the given hash, regardless of branches. (The pseudo-version is supposed to sort after every version — tagged or otherwise — that came before it, but before the next tag that a human might plausibly want to apply to the branch.)

In your example, you should expect a pseudo-version that sorts just after v0.2.2, since it is semantically higher than the other tags and is a parent of the current master (5aaa858).

Of course, bugs are possible (and, I suspect in this case, likely). 🙂

@leitzler
Copy link
Contributor

Nice, and two more questions for the same context:

  • Is it just annotated tags or should non-annotated tags be considered as well?
  • semantically-latest as in limited to major versions v0 and v1 right? Like the following example should give v1.0.1
* 91ba41 (HEAD -> master) ..
* b319ca (tag: v2.1.0) ..
* c49efa (tag: v1.0.1) ..
* 4baa0c (tag: v0.9.5) ..

@bcmills
Copy link
Contributor

bcmills commented Apr 24, 2019

Is it just annotated tags or should non-annotated tags be considered as well?

We don't go out of our way to distinguish. I suspect that we should consider both.
(If you have evidence that we are or are not already — especially cases where we should arguably doing something other than what we are — please do let us know.)

@bcmills
Copy link
Contributor

bcmills commented Apr 24, 2019

semantically-latest as in limited to major versions v0 and v1 right?

The major-version handling probably needs work.

If there is no go.mod file, then for tagged versions we normally allow a major-version mismatch but add a +incompatible suffix. It's not obvious to me what we should do for pseudo-versions under that condition.

If there is a go.mod file, then we currently do not allow a major-version mismatch (but perhaps we should in case of replacements; I'll file a separate issue for that). Whatever we do, we probably should not generate a pseudo-version with a particular major-version component if we would not allow that same major version for the same contents with an explicit tag.

@thepudds
Copy link
Contributor

thepudds commented Apr 25, 2019

@leitzler For your example in #27171 (comment), Bryan commented in #27171 (comment) on the expected result, but what do you currently see as the actual result for your specific example with Go 1.12 or tip?

In other words, with the current implementation, do you see a pseudo-version that sorts just after one of these, and if so, which one:

  • v0.2.2 - the most recent in creation date, that is merged into master
  • v0.2.1 - the most recent, that is tagged on master

(Possibly related to #31673 or #31287).

@leitzler
Copy link
Contributor

Go 1.12.4

# go version
go version go1.12.4 linux/amd64
# cd $(mktemp -d) && go mod init foo
go: creating new go.mod: module foo
# go get -v github.com/leitzler/tagtests
go: finding github.com/leitzler/tagtests v0.2.2
go: downloading github.com/leitzler/tagtests v0.2.2
go: extracting github.com/leitzler/tagtests v0.2.2
Fetching https://github.com?go-get=1
Parsing meta tags from https://github.com?go-get=1 (status code 200)
# cat go.mod
module foo

go 1.12

require github.com/leitzler/tagtests v0.2.2 // indirect
# cat go.sum
github.com/leitzler/tagtests v0.2.2 h1:iHmrV3vzYCN0qAo+SSyKUBtPQX4uvo65IAmo3Hh5Qw0=
github.com/leitzler/tagtests v0.2.2/go.mod h1:wK4dd9tQFb9oteDJOe6SYbpZj44o+r94PUWEw+ZFpg4=

@leitzler
Copy link
Contributor

However, if I ask for @master it will give me a pseudo version based on v0.2.1:

# go list -u -m all
foo
github.com/leitzler/tagtests v0.2.2
# go get -v github.com/leitzler/tagtests@master
go: finding github.com/leitzler/tagtests master
go: downloading github.com/leitzler/tagtests v0.2.2-0.20190424071028-5aaa858a59e2
go: extracting github.com/leitzler/tagtests v0.2.2-0.20190424071028-5aaa858a59e2
Fetching https://github.com?go-get=1
Parsing meta tags from https://github.com?go-get=1 (status code 200)
# go list -u -m all
foo
github.com/leitzler/tagtests v0.2.2-0.20190424071028-5aaa858a59e2 [v0.2.2]

@thepudds
Copy link
Contributor

thepudds commented Apr 25, 2019

@leitzler OK, Go 1.12 seems to give the wrong answer for @master in your simple example according to what Bryan said in #27171 (comment), which is perhaps the same issue as #31673.

In #27171 (comment) above, I had tried to differentiate between what seemed to be at least three different underlying problems for the different examples cited in this issue. It would at least seem from the outside that your example and #31673 are likely a fourth category of problem.

@gopherbot
Copy link

Change https://golang.org/cl/174061 mentions this issue: WIP: Change RecentTag in modfetch/git to use for-each-ref instead of describe

gopherbot pushed a commit that referenced this issue May 1, 2019
…tags and tags on other branches

Pseudoversion determination depends in part on the results from gitRepo.RecentTag, which currently invokes:

git describe --first-parent --always --abbrev=0 --match <prefix>v[0-9]*.[0-9]*.[0-9]* --tags <rev>

The comment at #27171 (comment) describes some problems with the current approach.

One problem is Docker and other repos can have tags that are not valid semver tags but that still match a glob pattern of v[0-9]*.[0-9]*.[0-9]* which are found by 'git describe' but then rejected by cmd/go, and hence those repos currently can end up with v0.0.0 pseudoversions instead of finding a proper semver tag to use as input to building a pseudoversion  (when then causes problems when the v0.0.0 pseudoversion is fed into MVS). An example problematic tag is a date-based tag such as 'v18.06.16', which matches the glob pattern, but is not a valid semver tag (due to the leading 0 in '06').

Issues #31673, #31287, and #27171 also describe problems where the '--first-parent' argument to 'git describe' cause the current approach to miss relevant semver tags that were created on a separate branch and then subsequently merged to master.

In #27171, Bryan described the base tag that is supposed to be used for pseudoversions as:

"It is intended to be the semantically-latest tag that appears on any commit that is a (transitive) parent of the commit with the given hash, regardless of branches. (The pseudo-version is supposed to sort after every version — tagged or otherwise — that came before it, but before the next tag that a human might plausibly want to apply to the branch.)"

This CL solves the glob problem and tags-on-other-branches problem more directly than the current approach: this CL gets the full list of tags that have been merged into the specific revision of interest, and then sorts and filters the results in cmd/go to select the semantically-latest valid semver tag.

Fixes #31673
Fixes #31287
Updates #27171

Change-Id: I7c3e6b46b2b21dd60562cf2893b6bd2afaae61d5
Reviewed-on: https://go-review.googlesource.com/c/go/+/174061
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/176417 mentions this issue: cmd/go: move two vcs test repos to vcs-test.golang.org

@thepudds
Copy link
Contributor

@rogpeppe does #27173 look related to your most recent report here?

@rogpeppe
Copy link
Contributor Author

@thepudds Yes, definitely related, but the description of that issue isn't very clear to me, so I'm not sure if it's a strict dupe.

The underlying problem here , I think, is that the go command is using existing information in $GOPATH to determine the version resolution, but instead it really needs to go to the source always, because the correct answer to the question "what is the latest version for this commit" can change over time.

@thepudds
Copy link
Contributor

Hi @rogpeppe, agreed not a strict dup. Part of the relationship is #27173 might show how one can put the wrong semver tag within a pseudoversion, and it still works without the go tool changing it to be a correct semver tag. That said, #27173 happens to not be 100% clear to me either.

The underlying problem here , I think, is that the go command is using existing information in $GOPATH to determine the version resolution, but instead it really needs to go to the source always, because the correct answer to the question "what is the latest version for this commit" can change over time.

There might be performance implications always of going to the source as part of something like go build or go list, etc.? You could say go get should always go to the source to verify any pseudoversions are correct, but go get foo@<version> mostly has the same behavior as editing the go.mod to read require foo <version> and then doing a go build or go test or similar, which means it might not make sense for go get either. However, maybe go mod tidy could be changed to always go to the source to verify any pseudoversion is correct, given it is already more-or-less documented to be more expensive in terms of module resolution, and go mod tidy is a rarer operation, and it is called tidy. But not 100% sure if that makes sense.

@bcmills
Copy link
Contributor

bcmills commented May 22, 2019

@rogpeppe, thanks for the clear reproducer. That confirms my suspicions about the mechanism of the bug.

I tend to agree that we should go to version control (or re-query the proxy) to resolve commits to versions. It's not clear to me whether we should try the local cache first when GOPROXY is set to off, although I suspect we should not.

@bcmills
Copy link
Contributor

bcmills commented May 22, 2019

@thepudds, there should be no performance implication for go build or go list in the steady state: once we have resolved the commit to a specific pseudo-version, we store that pseudo-version in the go.mod file and we can use that rather than re-resolving the underlying commit.

(And, in fact, it may be important not to re-resolve the version as the underlying tags evolve, since that can affect the MVS result and impact reproducibility.)

@rogpeppe
Copy link
Contributor Author

It's not clear to me whether we should try the local cache first when GOPROXY is set to off, although I suspect we should not.

I'd second this.

@thepudds
Copy link
Contributor

The underlying problem here , I think, is that the go command is using existing information in $GOPATH to determine the version resolution, but instead it really needs to go to the source always, because the correct answer to the question "what is the latest version for this commit" can change over time.

I think I misunderstood @rogpeppe's above suggestion then.

Based on @bcmills comment, is the suggestion instead:

  • always go back to source when doing a module query such as go get github.com/labstack/gommon@34167a09256a
  • but do not always go back to source when doing something like go get github.com/labstack/gommon@v0.0.0-20181219175904-34167a09256a (which is a syntactically valid pseudoversion, but has the "wrong" semver tag within it)?

If that is the suggestion, then I withdraw my comment #27171 (comment) on performance / correctness concern.

@thepudds
Copy link
Contributor

thepudds commented May 29, 2019

The pseudoversion for golang.org/x/text in #27171 (comment) seemed to be sensitive to what was in the cache. From looking at that briefly several months ago and mentioned in #27171 (comment), it seemed to be due to different results for the tag returned by git describe on later runs of go get.

Trying to recreate the problem from #27171 (comment) with current tip no longer seems to exhibit the problem:

go get golang.org/dl/gotip && gotip download
export GOPATH=$(mktemp -d)
cd $(mktemp -d)
gotip mod init tempmod
gotip list -m -json golang.org/x/text@master
gotip list -m -json golang.org/x/text@master
gotip list -m -json golang.org/x/text@master

which consistently returns:

{
        "Path": "golang.org/x/text",
        "Version": "v0.3.2",
        "Time": "2019-04-25T14:42:06-07:00"
}

Using the commit hash from the initial report in #27171 (comment) also now seems consistent:

export GOPATH=$(mktemp -d)
cd $(mktemp -d)
gotip mod init tempmod
gotip list -m -json golang.org/x/text@e6919f6577db
gotip list -m -json golang.org/x/text@e6919f6577db
gotip list -m -json golang.org/x/text@e6919f6577db

which consistently returns:

{
        "Path": "golang.org/x/text",
        "Version": "v0.3.1-0.20181227161524-e6919f6577db",
        "Time": "2018-12-27T08:15:24-08:00"
}

I am not sure if that is due to changes in the x/text repo, or other ecosystem changes, or perhaps due to changes in the go tool (e.g., maybe CL https://golang.org/cl/174061, which means it now no longer uses git describe in the section of code that was previously getting an unexpected tag for x/text).

@hyangah I would be curious to hear if you able to reproduce your original report from #27171 (comment), or perhaps a variation?

@rogpeppe
Copy link
Contributor Author

@thepudds I just confirmed that my reproducer in #27171 (comment) still shows this issue with Go tip (as of commit e883d00).

Perhaps you could check that to see if it works for you?

@hyangah
Copy link
Contributor

hyangah commented May 30, 2019

@thepudds I can't reproduce my report that uses golang.org/x/text either. I tested it again after manually reverting the cl/174061 and still couldn't reproduce the issue. So it's possible there was a different underlying issue, and now it's not reproducible.

@bcmills
Copy link
Contributor

bcmills commented May 30, 2019

golang.org/x/text got a new v0.3.2 tag recently, and it points to the same commit as the current master branch. That's probably why that example no longer reproduces.

@thepudds
Copy link
Contributor

thepudds commented May 30, 2019

In comment #27171 (comment) from several months ago, I had attempted to tease apart what the different underlying problems might be. At that time, the count was three.

At this point, there are probably at least 5-6 underlying problems reported in this issue, which I will attempt to enumerate as:

  1. The initial report by @rogpeppe that created this issue in cmd/go: go get $module@$commit should resolve it to the highest appropriate pseudo-version #27171 (comment).

    • Unclear if this was a "hangover" from vgo not doing the right thing, or maybe an early case of item 5 below ("cache pollution"), or maybe something else entirely.
  2. Docker and other repos that have tags that match a glob pattern of v[0-9]*.[0-9]*.[0-9]* can incorrectly end up with v0.0.0 pseudo-version.

  3. In cmd/go: pseudo-versions for post-v1 commits use the wrong major version #29262, rsc.io/quote at master has two different major versions following the Major Subdirectory approach, and go get rsc.io/quote@master requests a v1 version but does not find a v1 based semver tag because it first finds the v3 semver tag.

    • Not fixed.
  4. golang/xtext in cmd/go: go get $module@$commit should resolve it to the highest appropriate pseudo-version #27171 (comment) seems to get different results for the tag returned by git describe on later runs of go get.

  5. Tags on other branches were not handled properly (cmd/go: Improve module tag discovery #31673, cmd/go: specifying module branch writes incorrect version tag to go.mod #31287, plus related comments here from @leitzler and myself such as cmd/go: go get $module@$commit should resolve it to the highest appropriate pseudo-version #27171 (comment))

  6. Cache pollution where a bad pseudoversion lives on, as reported by @rogpeppe in cmd/go: go get $module@$commit should resolve it to the highest appropriate pseudo-version #27171 (comment).

    • Not fixed (and not yet fully investigated to my knowledge anyway).
  7. Arguable if cmd/go: unnecessarily multiple possible pseudo versions for one commit #27173 deserves to be in this list given it is still an open separate issue, but it might be related to "cache pollution" given that cmd/go: unnecessarily multiple possible pseudo versions for one commit #27173 issue illustrates pushing a series of "bad" psuedoversions into a go.mod that are then trusted and not updated to a more correct psuedoversion. That issue though also touches on other things, including that there might not be a single correct canonical answer for a pseudoversion.

    • Not fixed.

Given the variety above, it would not be shocking if there were additional underlying issues not yet teased out.

@gopherbot
Copy link

Change https://golang.org/cl/179857 mentions this issue: cmd/go/internal/modfetch: use the resolved version to search for tags in (*codeRepo).convert

gopherbot pushed a commit that referenced this issue May 31, 2019
… in (*codeRepo).convert

Previously, we used the passed-in statVers as the basis for tag search,
but it is not always valid.
Instead, use info.Name, which (by precondition) must be valid.

Updates #32161
Updates #27171

Change-Id: Iaecb5043bdf2fefd26fbe3f8e3714b07d22f580f
Reviewed-on: https://go-review.googlesource.com/c/go/+/179857
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/181881 mentions this issue: cmd/go: validate pseudo-versions against module paths and revision metadata

@gopherbot
Copy link

Change https://golang.org/cl/182178 mentions this issue: cmd/go/internal/modfetch: re-resolve commit hashes in readDiskStat

@bcmills
Copy link
Contributor

bcmills commented Jun 18, 2019

I've seen several examples in the module index (https://index.golang.org) of invalid pseudo-versions starting with v0.0.0-0.. If they originated with a go command bug it seems fine to allow them, but looking through the history of the code I'm not able to find any such bug.

Does anyone have any examples of a real go.mod file containing a v0.0.0-0.[…] version string in the wild? If so, do you know how it got there?

(Perhaps @marwan-at-work, @rogpeppe, @myitcv, or one of the other folks with third-party module-related tools has a clue?)

@myitcv
Copy link
Member

myitcv commented Jun 18, 2019

has a clue?

No clue here.

@leitzler
Copy link
Contributor

I noticed one of my repos in a listing you pasted on slack (or in another issue? Can’t really remember where I saw it). Anything regarding https://github.com/modsdemo is my fault. Have been playing around on purpose with it.

@bcmills
Copy link
Contributor

bcmills commented Jun 18, 2019

@leitzler, thanks for the heads-up. (The comment you saw it in was #27173 (comment).)

gopherbot pushed a commit that referenced this issue Jun 21, 2019
…tadata

Previously, most operations involving pseudo-versions allowed any
arbitrary combination of version string and date, and would resolve to
the underlying revision (typically a Git commit hash) as long as that
revision existed.

There are a number of problems with that approach:

• The pseudo-version participates in minimal version selection. If its
  version prefix is inaccurate, the pseudo-version may appear to have
  higher precedence that the releases that follow it, effectively
  “pinning” the module to that commit. For release tags, module
  authors are the ones who make the decision about release tagging;
  they should also have control over the pseudo-version precedence
  within their module.

• The commit date within the pseudo-version provides a total order
  among pseudo-versions. If it is not accurate, the pseudo-version
  will sort into the wrong place relative to other commits with the
  same version prefix.

To address those problems, this change restricts the pseudo-versions
that the 'go' command accepts, rendering some previously
accepted-but-not-canonical versions invalid. A pseudo-version is now
valid only if all of:

1. The tag from which the pseudo-version derives points to the named
   revision or one of its ancestors as reported by the underlying VCS
   tool, or the pseudo-version is not derived from any tag (that is,
   has a "vX.0.0-" prefix before the date string and uses the lowest
   major version appropriate to the module path).

2. The date string within the pseudo-version matches the UTC timestamp
   of the revision as reported by the underlying VCS tool.

3. The short name of the revision within the pseudo-version (such as a
   Git hash prefix) is the same as the short name reported by the
   underlying cmd/go/internal/modfetch/codehost.Repo. Specifically, if
   the short name is a SHA-1 prefix, it must use the same number of
   hex digits (12) as codehost.ShortenSHA1.

4. The pseudo-version includes a '+incompatible' suffix only if it is
   needed for the corresponding major version, and only if the
   underlying module does not have a go.mod file.

We believe that all releases of the 'go' tool have generated
pseudo-versions that meet these constraints. However, a few
pseudo-versions edited by hand or generated by third-party tools do
not. If we discover invalid-but-benign pseudo-versions in widely-used
existing dependencies, we may choose to add a whitelist for those
specific path/version combinations.

―

To work around invalid dependencies in leaf modules, users may add a
'replace' directive from the invalid version to its valid equivalent.
Note that the go command's go.mod parser automatically resolves commit
hashes found in 'replace' directives to the appropriate
pseudo-versions, so in most cases one can write something like:

	replace github.com/docker/docker v1.14.0-0.20190319215453-e7b5f7dbe98c => github.com/docker/docker e7b5f7dbe98c

and then run any 'go' command (such as 'go list' or 'go mod tidy') to
resolve it to an appropriate pseudo-version. Note that the invalid
version will still be used in minimal version selection, so this use
of 'replace' directives is an incomplete workaround.

―

One of the common use cases for higher-than-tagged pseudo-versions is
for projects that do parallel development on release branches. For
example, if a project cuts a 'v1.2' release branch at v1.2.0, they may
want future commits on the main branch to show up as pre-releases for
v1.3.0 rather than for v1.2.1 — especially if v1.2.1 is already tagged
on the release branch. (On the other hand, a backport of a patch to
the v1.2 branch should not show up as a pre-release for v1.3.0.)

To address this use-case, module authors can make use of our existing
support for pseudo-versions derived from pre-release tags: if the
author adds an explicit pre-release tag (such as 'v1.3.0-devel') to
the first commit after the branch, then the pseudo-versions for that
commit and its descendents will be derived from that tag and will sort
appropriately in version selection.

―

Updates #27171
Fixes #29262
Fixes #27173
Fixes #32662
Fixes #32695

Change-Id: I0d50a538b6fdb0d3080aca9c9c3df1040da1b329
Reviewed-on: https://go-review.googlesource.com/c/go/+/181881
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@golang golang locked and limited conversation to collaborators Jun 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

7 participants