@gopherbot, please add label modules

Hi @markbates, I'm not 100% sure this is the root cause, but given 'pop' is >= v2, I believe you need to have the major version at the end of the module path in your module directive in your go.mod file.

Right now, it shows:

module github.com/gobuffalo/pop

https://github.com/gobuffalo/pop/blob/d38e78212cc54cd9317a67522137701a3f40af82/go.mod

This is covered a bit on the "Modules" wiki here:

https://github.com/golang/go/wiki/Modules#semantic-import-versioning

Could you try appending the major version to the end of the module path in the module directive in the go.mod file? I think that would mean appending /v4 if I've followed.

Yes, but as noted in this blog post, https://blog.gobuffalo.io/the-road-to-1-0-go-modules-c55b42af70de, doing so would cause breakage for lots of people. I would rather not create a release that would potentially harm users to something like that.

According to @spf13 the Semantic Import Versioning rule has been relaxed, so it would make sense that this would work.

The logic doesn't make sense. For packages that are already about v2 can't use Go Modules as it currently stands, or else Go Modules will ignore them. That is definitely not the logical workflow. I can use v4.5.9 because it doesn't have a go.mod file, but I can't use v4.6.4 because it does.

Also, at the risk of getting ahead of ourselves, if that was the issue that was biting you, I would suggest re-purposing this issue into how it might have been easier to diagnose the issue (e.g., with a warning, or with -v output from 'go get', or ___).

edit: Sorry, got ahead of things with this comment. Please ignore this particular comment for now.

Hi @markbates, I'm not sure specifically what @spf13 was referencing, but perhaps he was describing the work where Go versions 1.9.7+ and 1.10.3+ have been updated to know how to properly interpret a /v2 or higher that appears in an import path, for example:

https://go-review.googlesource.com/c/go/+/109340

As far as I am aware, that would still requiring having a /v4 at the end of the module directive in the go.mod file for 'pop' given the major version of the 'pop' module is v4.

If you are looking to still support Go 1.8.x with a v4.x.x module, then the only alternatives I am aware of are:

• creating a /v4 subdirectory, place a new go.mod within the subdirectory, append /v4 to the module path, and copy or perhaps move the code into the subdirectory (as described in the initial vgo blog posts, and also a bit on the Modules wiki link above), or
• waiting until 1.9.7 and 1.10.3 have been out long enough to be a reasonable choice for your users

All that said, I'm just sharing my current personal understanding... I'm happy to learn more myself, and I certainly don't want to mis-direct you, so please take with a grain of salt.

@thepudds All of that has been relaxed because it presents a chicken and egg problem for a lot of >2 repos out there.

The logic here is either coded incorrectly, which is what I think it is (a left over from earlier stricter releases), or the logic is fundamentally flawed, which I don't think it is. Not pulling perfectly good releases when searching for "latest" because they contain a go.mod makes zero sense. It's not what one would expect the behavior to be, nor does it encourage people to actively start using Go Modules as they'll break any >2 repo that adds them.

/cc @rsc

Hi @markbates, and just to confirm, part of the current goal is to continue to support people using Go 1.8.x? I see on the Buffalo site:

Buffalo has a minimum Go dependency of 1.8.1.

I'm guessing the intent is to continue to support 1.8.x right now, but I thought worth asking.

We’re moving to 1.9 in the next release.

And just to be clear, this has zero to do with Buffalo. I’m just using that repo as an example. Please don’t get hung up on that.

This is about the logic of finding the “latest” version of a v2x repo.

Is it acceptable to require version 1.9.7+ for buffalo?

As far as I am aware, that it the oldest release that picks up the minimal module awareness that makes older releases able to most easily consume a v2+ module.

Please stop making this about Buffalo. This is about the logic of the resolver.

Hi @markbates, sorry, we posted at close to the same time.

Hi @markbates, I'll summarize what I was trying to say, and then leave it to others that are more knowledgeable.

My (possibly incorrect) understanding:

• If there is a properly constructed go.mod file at the root of the tree of Go source files for a package foo, then Go versions 1.9.7+, 1.10.3+, and 1.11 know how to properly interpret a /v2 or /v4 that appears within an import path when importing foo (by interpreting the /v2 or v4 as being a major version).
• But regardless of whether the consumer is Go 1.9.7+, 1.10.3+, 1.11 or vgo, a properly constructed go.mod file for a v4.x.x module includes appending /v4 to the end of the module path in the module directive in the module's go.mod file.
• And perhaps not as relevant here, but Go versions prior to Go 1.9.7 and 1.10.3 have a much more awkward situation regarding consuming a v2 or higher module.

All that said, my understanding might be stale, or perhaps was just wrong to begin with. ;-)

Hi Mark,

Sorry for all the confusion. Lots going on here. It's going to take a while for everyone to get up to speed and for a good variety of intro docs to be written.

First, it appears you think you can opt out of semantic import versioning. You cannot. If you're using modules, you must use semantic import versioning.

• github.com/gobuffalo/pop is the module path and import path for the v0/v1 series of versions in the gobuffalo/pop repo.
• github.com/gobuffalo/pop/v2 is the module path and import path for the v2 series.
• github.com/gobuffalo/pop/v3 is the module path and import path for the v3 series.
• github.com/gobuffalo/pop/v4 is the module path and import path for the v4 series.

Normally, if the go command sees a tag v4.0.0 in the gobuffalo/pop repo, it expects that tag to be the github.com/gobuffalo/pop/v4 module.

As an exception, to help with repos tagged before semantic import versioning, if the go command sees a tag >= v2.0.0 but the tagged tree has no go.mod, then the go command puts that version into the v0/v1 series - that is, it lets it be a version for github.com/gobuffalo/pop - but with a +incompatible suffix. go help modules explains:

Code written before the semantic import versioning convention
was introduced may use major versions v2 and later to describe
the same set of unversioned import paths as used in v0 and v1.
To accommodate such code, if a source code repository has a
v2.0.0 or later tag for a file tree with no go.mod, the version is
considered to be part of the v1 module's available versions
and is given an +incompatible suffix when converted to a module
version, as in v2.0.0+incompatible. The +incompatible tag is also
applied to pseudo-versions derived from such versions, as in
v2.0.1-0.yyyymmddhhmmss-abcdefabcdef+incompatible.

This exception is what made it possible for go get github.com/gobuffalo/pop to consider the v4.5.9 tag part of the v0/v1 series. It called the version v4.5.9+incompatible instead of v4.5.9.

You can see all the versions by using:

$ go list -m -versions -f '{{join .Versions "\n"}}' github.com/gobuffalo/pop@latest
v3.3.0+incompatible
v3.3.1+incompatible
...
v4.5.8+incompatible
v4.5.9+incompatible
v4.5.11+incompatible
$ 

But when you create and tag a version with a go.mod, that's interpreted as a signal that the repo is aware of modules and therefore aware of semantic import versioning, so if v4.5.10 has a go.mod, then it should be saying module github.com/gobuffalo/pop/v4 and the code should be importing directories in that repo using paths with pop/v4 in them. That's why all those later tags are not seen - now that there's a go.mod in the repo, the expectation is that those tags are tags for the module github.com/gobuffalo/pop/v4, not plain pop (v1).

As @thepudds points out, pop can move to the /v4/ paths and older go commands (1.9.7+ and 1.10.3+) will do the right thing. But not 1.8 (which is officially unsupported at this point, since 1.10 came out).

edit: The context for this comment here is that semantic import versioning means code that opts into modules must include the major version in the import path for any imported v2+ modules. However, a natural question is then how does that interact with other code in the same build if that other code has not yet opted in to modules. The short version is the go command "does the right thing" when using Go 1.9.7+, 1.10.3+ or 1.11, including old code that has not opted into modules does not need to be updated to include the major version in import paths when consuming v2+ modules. The remainder of this comment attempts to describe that in more detail, including via a runnable example that includes (in a single build) a shared v2+ module being imported both by code that has opted in to modules as well as by code that has not yet opted in to modules.

The Buffalo blog post mentioned above includes the following snippet as part of the problem statement:

I have library foo/bar that is currently at 3.7.0 that library is currently being used both in Buffalo and imported into the end user’s code as well, for their use.

If we add a go.mod file we can set the name of the module as foo/bar/v3. Now, if we update to use that import path, and we’re on Go 1.11 Buffalo will build successfully.

The problem comes in for those who aren’t yet using the Go modules functionality. If they do a go get on Buffalo it will fail to compile because it can’t find the v3 import.

It sounds like that problem is not the only modules-related issue confronting Buffalo, but I wanted to share a quick runnable example that tries to show an illustration of that particular issue (although using only three trivial packages).

I think this example ends up showing:

• different packages in the overall Go ecosystem can opt-in at different rates to modules and semantic import versioning, and
• when building with 1.9.7+, 1.10.3+ or 1.11, then even in a single build:
  • different consumers of a shared package are not required to opt-in simultaneously to modules and semantic import versioning
  • 'old' code can still use the 'old' import path for a v2+ module while 'new' code is using the new /vN import path
• and that's true in this example without requiring anyone to do any gymnastics like copying or moving files into a subdirectory

Perhaps this example below helps makes an aspect of the conversation here more concrete, including for any others reading this issue. (Or perhaps this example is not on target, in which case I will likely learn something myself).

In any event, the more concrete example is:

• example.com/scratchpad/oldhello.go is the entirety of the simple top-level package (main) in the build:
  • we could suppose I wrote it 18 months ago and have no desire to update it any time soon
  • oldhello has not opted in to semantic import versioning
  • oldhello imports two packages: a and b
• For packages a and b:
  • both are v3.0.x
  • both have opted in to semantic import versioning
  • both have go.mod files
• Package a also imports package b (and hence a must use /v3 in the import path when importing b)

So at first glance it seems we could be in trouble, because in a single build:

• oldhello.go and a are both importing b
• oldhello.go wants to import package b without using /v3 in the import path for b
• package a needs to import package b with /v3 in the import path for b

However, my understanding is that potential issue is automatically resolved if using 1.9.7+, 1.10.3+, or 1.11, because the go tooling has been taught how to automatically handle that and let a build straddle a pre-module world and post-module world (and it does so without requiring oldhello.go to change, and while allowing a and b to relatively easily opt in to semantic import versioning ahead of their client oldhello.go).

oldhello.go

Here is oldhello.go. This 'old' package does not have a go.mod, and note how it does not use /v3 to import a or b (even though a and b are both v3+ modules):

$ cd $GOPATH/src/example.com/sratchpad/oldhello
$ ls
oldhello.go

$ cat oldhello.go

package main

import (
	"github.com/thepudds/example-package-a"   
	"github.com/thepudds/example-package-b"   
)

func main() {
	a.Hello()
	b.Hello()
}

Package a

Here is package a, which also imports package b. Both a and b opted in to semantic import versioning. The key point is that a does use /v3 in the import path for b.

a.go: https://github.com/thepudds/example-package-a/blob/v3.0.1/a.go

Also, a declares itself to be v3 in its own go.mod.

go.mod: https://github.com/thepudds/example-package-a/blob/v3.0.1/go.mod

Running example with Go 1.9.7 (works)

Here is running oldhello.go using Go 1.9.7. It works despite the mismatch regarding /v3 in the import path for b because 1.9.7+ has been taught how to handle this situation:

# first get go1.9.7 if we don't have it:
$ go get golang.org/dl/go1.9.7
$ go1.9.7 download

# now using 1.9.7,  'go get' a and b then build oldhello, all of which works:
$ cd $GOPATH/src/example.com/sratchpad/oldhello
$ go1.9.7 get -u github.com/thepudds/example-package-a
$ go1.9.7 get -u github.com/thepudds/example-package-b
$ go1.9.7 build .

# running generates the following output:
$ ./oldhello
Hello from package a (which is on v3.0.1 and has opted-in to semantic import versioning)
Hello from package b (which is on v3.0.3 and has opted-in to semantic import versioning)
Hello from package b (which is on v3.0.3 and has opted-in to semantic import versioning)

Running example with Go 1.8 (fails, as expected)

In contrast, here is building oldhello.go using Go 1.8. As expected, that fails (because of the mismatch with /v3 in the import path for b, and Go 1.8 has not been taught how to handle this situation):

$ go get golang.org/dl/go1.8.6
$ go1.8.6 download
$ cd $GOPATH/src/example.com/sratchpad/oldhello
$ go1.8.6 build .

.../src/github.com/thepudds/example-package-a/a.go:6:2: 
    cannot find package "github.com/thepudds/example-package-b/v3" in any of:
        .../sdk/go1.8.6/src/github.com/thepudds/example-package-b/v3 (from $GOROOT)
        .../src/github.com/thepudds/example-package-b/v3 (from $GOPATH)

Summary

For the piece of the problem statement in the blog snippet at the top of this comment from the Buffalo blog:

• As far as I'm aware, Go 1.9.7+, 1.10.3+ and 1.11 seem to handle it fine.
• But it is indeed an issue if using Go 1.8, for example.

All that said, this is just a simple example intended to help the conversation. Sorry for the length.

@rsc & @bcmills I had a chat with @markbates about this and it's clear to me from this that there is an issue here that needs to be addressed. It's not that there is a problem with Go modules, per say, rather it's a flaw in our education. There are people currently publishing versions incorrectly and it's becoming an issue.

Another example is https://github.com/gofrs/uuid which added a go.mod file in the tag 3.1.1 with module "github.com/gofrs/uuid/v3". The reason this is an issue is that there are now 4 possibilities when importing this package and each of them results in a different set of source being imported:

You import "github.com/gofrs/uuid" and you are using modules → 3.1.0
You import "github.com/gofrs/uuid" and you are not using modules → master
You import "github.com/gofrs/uuid/v3" and you are using modules → 3.1.2
You import "github.com/gofrs/uuid/v3" and you are not using modules → breaks

@spf13 I did some investigation (see gofrs/uuid#61 (comment)), and found that:

1. We seem to have undercommunicated the fact that changing the import path is a breaking change, or the fact that adding a module definiton to a v2+ repo changes the import path, or both.

2. There is a workaround available to module owners, although it is not well-published yet: a forwarding package without a go.mod file at v3.2.0, forwarding to the /v4 module, would work with any compatible v4 for all importers using Go 1.9.7 and up, including those using Go 1.11 in module mode. (That should work with both the major-branch and major-subdirectory style of module layout.)

@spf13 @bcmills

Building on Bryan's comment -- one other aspect that might have been undercommunicated is "minimal module awareness", and the exact behavior you get in different scenarios with the backported behavior in Go versions 1.9.7+ and 1.10.3+, and also how it all interacts with Go 1.11 (including with the different possible settings of GO111MODULE environment variable) for code that has opted in to modules, or code that has not opted in to modules, or mixed scenarios.

On the surface, it can seem like the "minimal module awareness" is simple, but I think I've seen very savvy people (including perhaps some even within the core Go team?) go through a mental process regarding how "minimal module awareness" works (and how to use it) that might be something along the lines of:

1. "Oh, that sounds simple; more backwards compatibility is good; it just works", to
2. "Hmm, I'm not sure if it works the way I thought it did", to
3. "Wait, there's no way this could work at all in any form"

At least, I know I went through some confusion on the topic.

When you look at the technical details discussed in this issue here as well as in gofrs/uuid#61, I think the exact behavior of "minimal module awareness" plays into at least parts of it.

It might be that "minimal module awareness" was one of the biggest improvements to vgo based on community feedback during the prototype phase, but that also means it doesn't feature in the extensive introductory blog posts on vgo (I think).

On the one hand "minimal module awareness" is just a transitional measure, but on the other hand, we are very much in a transitional phase right now and it seems it is playing an important role in practice right now.

In any event, just sharing an observation on where some early adopters might have hit some snags...

Change https://golang.org/cl/162699 mentions this issue: cmd/go: only generate a go.mod file during 'go mod init'