Quoting https://research.swtch.com/vgo-module:

We want to encourage more developers to tag releases of their packages, instead of expecting that users will just pick a commit hash that looks good to them. Tagging explicit releases makes clear what is expected to be useful to others and what is still under development. At the same time, it must still be possible—although maybe not convenient—to request specific commits.

We want to move away from invoking version control tools such as bzr, fossil, git, hg, and svn to download source code. These fragment the ecosystem: packages developed using Bazaar or Fossil, for example, are effectively unavailable to users who cannot or choose not to install these tools. The version control tools have also been a source of exciting security problems. It would be good to move them outside the security perimeter.

Can't vgo leverage the already existing v2.0+ git tags released on Github? Wouldn't it "just work" for packages that don't have any complex transitive dependencies?

I just hope that migrating monorepos from golang/dep to vgo (can we expect a migration tool, right?) will be very straight-forward. I hope we won't be blocked by some v2.0+ packages that "just work" but are not really maintained anymore and thus "not ready" for vgo.

Any guidance / feedback appreciated. Thanks!

Have a look at #25810 (comment)

@VojtechVitek sorry, I was away from keyboard when I sent that last message, hence it was brief.

Here is a complete example that I previously posted to #vgo on Slack. I'll close this issue because I think it answers your question but please shout if not.

Create a module using the github.com/go-chi/chi example:

$ mkdir hello
$ cd hello
$ vgo mod -init -module example.com/hello
vgo: creating new go.mod: module example.com/hello
$ cat <<EOD >hello.go
package main

import (
        "net/http"
        "github.com/go-chi/chi"
)

func main() {
        r := chi.NewRouter()
        r.Get("/", func(w http.ResponseWriter, r *http.Request) {
                w.Write([]byte("welcome"))
        })
        http.ListenAndServe(":3000", r)
}
EOD

Now because, at the time of writing, github.com/go-chi/chi:

• has a major version >= 2
• has not been converted to a Go (vgo) module
• we want to use v3.3.2

we need to vgo get that specific version, which will be retrieved as a v0.0.0 psuedo version:

$ vgo get github.com/go-chi/chi@v3.3.2
vgo: finding github.com/go-chi/chi v3.3.2
vgo: downloading github.com/go-chi/chi v0.0.0-20171222161133-e83ac2304db3

Now do a build to check all is good:

$ vgo build

And check the contents of go.mod:

$ cat go.mod
module example.com/hello

require github.com/go-chi/chi v0.0.0-20171222161133-e83ac2304db3

Hi @myitcv,
thanks for your guidance. I think I understand now - your workaround should work.

we need to vgo get that specific version, which will be retrieved as a v0.0.0 psuedo version:

This is super confusing to me and I think it will cause a lot of confusion in the Go community.

Migrating from golang/dep, I will have to start using some v.0.0.0- pseudo versions instead of v3.3.2, just because vgo can't work properly with git tags.

Well, vgo list can:

$ vgo list -t github.com/go-chi/chi/v3
github.com/go-chi/chi/v3
	v3.0.0
	v3.1.0
	v3.1.1
	v3.1.2
	v3.1.3
	v3.1.4
	v3.1.5
	v3.2.0
	v3.2.1
	v3.3.0
	v3.3.1
	v3.3.2

But other vgo command cannot. The version gets "reverted" to v.0.0.0-*, since the upstream didn't adapt to the new rules (go.mod).

Maybe there's no workaround for this from the vgo's perspective. But in that case, would it make sense to provide some guidelines for the v2.0+ pkg authors and consumers to avoid any confusion?

I'm not the only one not getting it right away, see go-chi/chi#302.

cc @rsc

This is super confusing to me and I think it will cause a lot of confusion in the Go community.

Just to be clear, this is only required under the conditions listed in the bullet points that preceded it (reordered):

• has not been converted to a Go (vgo) module
• has a major version >= 2
• we want to use v3.3.2

As soon as the project is converted to a module, then you're set - no pseudo-versions required. Because when chi is converted to a module, the import path becomes github.com/go-chi/chi/v3. Your example would therefore become:

package main

import (
        "net/http"
        "github.com/go-chi/chi/v3"
)

func main() {
        r := chi.NewRouter()
        r.Get("/", func(w http.ResponseWriter, r *http.Request) {
                w.Write([]byte("welcome"))
        })
        http.ListenAndServe(":3000", r)
}

and your go.mod would look like:

module example.com/hello

require github.com/go-chi/chi v3.3.2

As was observed in go-chi/chi#302 (comment):

All internal references to go-chi would have to be changed to use a v3 import style.

Therefore, any "old" go (i.e. non vgo) users of the now-converted chi project would be fine so long as they use either 1.9.7 or 1.10.3.

Therefore, any "old" go (i.e. non vgo) users of the now-converted chi project would be fine so long as they use either 1.9.7 or 1.10.3.

Hmm, it's unlikely that upstream v2.0+ packages will stop supporting go 1.10 anytime soon by changing the internal import paths to "/v2" etc.

Thanks for clarification!

@myitcv Paul, would you mind taking a look at go-chi/chi#327 (review) and advice what should be done in order for project authors to support both new (vgo) and old (module unaware) versions of Go

assuming the project

has not been converted to a Go (vgo) module
has a major version >= 2
we want to use v3.3.2

Much appreciated!

@VojtechVitek how many Go versions do you need to support? With 1.9.7 and 1.10.3 now out, any project supporting up to the two last Go versions (which I think is a majority) should be able to switch to Go modules soon.

The only potential problem I see is people still using old minor releases like 1.10.2. This is why I'm holding off for a few weeks before porting my v2+ projects to Go modules. I'll probably do the switch once 1.11 is out, at which point backported support will have been out for two full months.

If you need to support the latest three major Go versions, waiting until 1.11 is out should also do the trick. This obviously doesn't scale past three major Go versions, but hopefully the set of v2+ packages with that constraint is very small.

@VojtechVitek as @mvdan points out, if you are happy only supporting Go 1.9.7 (or later) and 1.10.3 (or later) from the 1.9.x and 1.10.x series respectively, and of course Go 1.11 and later, and your project is v >= 2, then you can safely convert your project to be a Go module. Otherwise you will hit issues with older Go versions not being able to handle the /vX in import paths.

I'm not happy dropping support for Go <=1.10.2, <=1.9.6 and 1.8.x for a long long time. I'm talking year+, maybe until 1.12 is out.

Go was always about backwards compatibility ... and now we just break stuff because vgo can't support the existing git tags properly .. and requires new "module" notation instead :(

Are there particular cases of users not being able to upgrade bugfix/minor versions? If I was maintaining a large open-source project and a bug was reported with Go 1.9.0, my first response would be for them to try with the latest 1.9.x.

This is how I understand Go supports older versions. For example, when it is said that the last stable release recieves important bugfixes, that only concerns the latest bugfix version. I wouldn't consider 1.9.0 maintained at this point.

Having said that, I understand that it's frustrating that Go modules don't play nice with v2+ projects. Although, I'd note that this is not a breaking change. You are not forced to start using a go.mod file :)

Are there particular cases of users not being able to upgrade bugfix/minor versions?

Go 1.10.2 was released 2 months ago as a stable version. We can't just suddenly break the stable builds "just because of vgo support".

Personally, I can upgrade few monorepos and I'm done. But not everyone is that lucky.. they just want to upgrade their dependencies to the latest minor/bugfix version to get bug fixes and security patches. That's it. They don't want to worry about some "hidden" difference between Go 1.10.2 and 1.10.3 and spend hours debugging the issue.

Go 1.9.0, my first response would be for them to try with the latest 1.9.x.

Sorry, but that sounds wrong. Backward compatibility is Go's promise, one of the pillars of the language. Any version in the 1.9.x range should behave the same. Quoting from https://blog.golang.org/versioning-proposal:

We believe that the promise of compatibility made developers feel much more comfortable relying on Go for production use and is a key reason that Go is popular today

Having said that, I understand that it's frustrating that Go modules don't play nice with v2+ projects. Although, I'd note that this is not a breaking change. You are not forced to start using a go.mod file :)

Yes, we're not forced to do so.. as long as everyone in the community understands they should use v0.0.0-DATE-HASH for the time being for existing Github projects of version v2.0+. Because vgo.

Why am I bringing this up? I'm hoping someone will explain the best practices, a guideline for both package authors & consumers. I'm hoping Go will stay stable for the sake of the language and the community.

I'm hoping someone will explain the best practices, a guideline for both package authors & consumers.

As I said before, I'm also on the same boat with a couple of v2+ projects, and I too would like there to be a "best practices" or "transition guide" document published. All I am saying is that I don't consider this a breaking change - it feels like more of a temporary annoyance for part of the libs out there.

Any version in the 1.9.x range should behave the same.

Depends on what you mean by "same" :) The only promise made is backwards compatibility. There is no promise that goes forward. If your package works with 1.9.5, there's no guarantee that it will work with 1.9.1.

Of course, it is possible for packages to sasy "we support all 1.9.x versions". My point was that, in practice, most projects out there only test against the latest 1.9.x, hence 1.9.0 is in a way unsupported by much of the community.

Yes, we're not forced to do so.. as long as everyone in the community understands they should use v0.0.0-DATE-HASH for the time being for existing Github projects of version v2.0+.

This is not strictly true. As outlined above the command is actually:

vgo get github.com/go-chi/chi@v3.3.2

The command refers to a specific, released version. vgo then does the lifting required. I also note #25898 at this point.

I'm hoping someone will explain the best practices, a guideline for both package authors & consumers. I'm hoping Go will stay stable for the sake of the language and the community.

There is absolutely nothing that requires you to make chi a module. But let's assume you elected to wait for Go 1.12 in order to convert it to a module (further assuming this is when vgo ceases to be "experimental"). In the interim, i.e. for the duration of Go 1.11 pre Go 1.12, those people using vgo (which won't be everyone because it will be opt-in) would have to follow the approach outlined above in order to use chi. When chi is made into a module, the transition would be trivial, and likely automatic (because vgo will have gained extra "fix" powers by then).

This feels like a totally satisfactory story for chi and other projects v >= 2 that choose, quite fairly, to support older versions of Go.

Just to note, vgo as a command will disappear to be "replaced" by go, driven by an environment variable for Go 1.11

/cc @rsc @bcmills in case there's anything they'd like to change/add.

Regarding:

assuming the project

has not been converted to a Go (vgo) module
has a major version >= 2
we want to use v3.3.2

And the concern:

Migrating from golang/dep, I will have to start using some v.0.0.0- pseudo versions instead of v3.3.2, just because vgo can't work properly with git tags.

I think that has changed recently in CL 124384?

Here is a snippet from the CL:

Repos written before the introduction of semantic import versioning
introduced tags like v2.0.0, v3.0.0, and so on, expecting that
(1) the import path would remain unchanged, and perhaps also
(2) there would be at most one copy of the package in a build.

We've always accommodated these by mapping them into the
v0/v1 version range, so that if you ran

go get k8s.io/client-go@v8.0.0

it would not complain about v8.x.x being a non-v1 version and
instead would map that version to a pseudo-version in go.mod:

require k8s.io/client-go v0.0.0-20180628043050-7d04d0e2a0a1

The pseudo-version fails to capture two important facts: first,
that this really is the v8.0.0 tag, and second, that it should be
preferred over any earlier v1 tags.

<...snip...>

This CL introduces a new version suffix +incompatible that
indicates that the tag should be considered an (incompatible)
extension of the v1 version sequence instead of part of its
own major version with its own versioned module path.
The requirement above can now be written:

require k8s.io/client-go v8.0.0+incompatible

(The +metadata suffix is a standard part of semantic versioning,
and that suffix is ignored when comparing two versions for
precedence or equality.

And here is a related snippet from the latest 'go help modules' (from tip today):

In semantic versioning, major version v0 is for initial development,
indicating no expectations of stability or backwards compatibility.
Major version v0 does not appear in the module path, because those
versions are preparation for v1.0.0, and v1 does not appear in the
module path either.

Code written before the semantic import versioning convention
was introduced may use major versions v2 and later to describe
the same set of unversioned import paths as used in v0 and v1.
To accommodate such code, if a source code repository has a
v2.0.0 or later tag for a file tree with no go.mod, the version is
considered to be part of the v1 module's available versions
and is given an +incompatible suffix when converted to a module
version, as in v2.0.0+incompatible. The +incompatible tag is also
applied to pseudo-versions derived from such versions, as in
v2.0.1-0.yyyymmddhhmmss-abcdefabcdef+incompatible.

So if I am following this particular issue, it sounds like if go-chi does not currently have have a go.mod file but is already at v3.3.2 tag, the require for your 'hello.go' example would no longer end up being a v0.0.0 pseudo-version (which was one of the concerns raised above by @VojtechVitek), and instead would end up being a slightly friendlier:

require github.com/go-chi/chi v3.3.2+incompatible

which is friendlier than the older behavior of ending up with something like:

require github.com/go-chi/chi v0.0.0-20171222161133-e83ac2304db3

Here is the new behavior in slightly more detail:

$ mkdir hello
$ cd hello
$ go mod -init -module example.com/hello

$ cat <<EOD >hello.go
package main

import (
        "net/http"
        "github.com/go-chi/chi"
)

func main() {
        r := chi.NewRouter()
        r.Get("/", func(w http.ResponseWriter, r *http.Request) {
                w.Write([]byte("welcome"))
        })
        http.ListenAndServe(":3000", r)
}
EOD

$ go get github.com/go-chi/chi@v3.3.2
$ go build

And the go.mod ends up after the build as:

$ cat go.mod 

module example.com/hello

require github.com/go-chi/chi v3.3.2+incompatible

Just a small remark, you even don't need to run go get github.com/go-chi/chi@v3.3.2 explicitly.
The new behaviour lets you just run go build without explicit versions, the latest semver tag will be resolved automatically.

cd `mktemp -d`
go mod -init -module example.com/hello
cat <<EOD >hello.go
package main

import (
        "github.com/go-chi/chi"
)

func main() {
        _ = chi.NewRouter()
}
EOD
go build

go.mod ends up as:

$ cat go.mod
module example.com/hello

require github.com/go-chi/chi v3.3.2+incompatible

Hi @mwf, thanks, and good clarification. I had thought part of the example cited here was to specifically use v3.3.2, but given v3.3.2 is currently the latest semver tag for go-chi your comment makes sense (and the original issue reported here was more about "how can pre-existing v2+ packages work at all?"', vs. the "we want to use v3.3.2" was added as a later example).

@thepudds @mwf Thanks for letting us know about the new +incompatible metadata flag! Very useful. I'm glad it's supported now.

So, as far as I understand, the github.com/go-chi/chi project needs to stick with v3.x+incompatible version for now. We can't release go.mod and thus release a new github.com/go-chi/chi/v3 import path, until we're comfortable with breaking builds for Go versions that are not aware of Go modules (pre- 1.9.7 and 1.10.3).

Hi @VojtechVitek

That sounds like a very reasonable choice for the go-chi project.

I still feel, though, that the more general original issue opened here isn't fully resolved in terms of wanting advice for maintainers of existing v2+ projects.

At the risk of writing something wildly wrong, I am going to take an initial attempt at a more comprehensive answer, and I'm hoping someone like @mwf, @myitcv, @mvdan, or @bcmills can jump in with corrections. (In general, there has been a large amount of information written about vgo/versioned modules, but it has also been evolving and tough to track the various changes and what the current advice might be).

Apologies in advance if this just ends up being noise, and this could certainly be written better and more succinctly, but in the interests of providing some grist for the mill and trying to move the ball forward, here is a rough draft:

Go 1.11 Versioned Modules: Migration and Compatibility Considerations (DRAFT)

The Go contributors are focused on providing a smooth migration from prior dependency managers and prior Go versions.

Part of the strategy is to allow for phased adoption. Different projects are expected to opt-in at different paces, and packages defined as Go 1.11 modules can be consumed by pre-1.11 Go toolsets if the proper strategies are followed.

General migration strategies for v0.x.x, v1.x.x, and v2+ projects

Some strategies for migrating to Go 1.11 modules and supporting older Go versions include:

1. go mod -init automatically translates the configuration from dep, glide, godep and 6 other pre-existing dependency managers into a go.mod file that produces the equivalent build.
2. You can use go mod -vendor to support older Go versions:
   • For the purpose of interoperating with older versions of Go, go mod -vendorcreates a directory named vendor in the root directory of the current module and stores there all the packages from dependency modules that are needed to support builds in the main module, with the selected versions corresponding to the go.mod file.
   • A go get issued by an older Go toolset will therefore use the module's vendor directory containing the proper versions of dependencies, even if that older Go toolset does not yet understand modules.
3. You can consider maintaining a go.mod in parallel to your prior dependency management solution and committing both sets of configuration to your repository during the transitional period.
4. You can consider creating a go.mod on-the-fly in CI** while treating your current dependency manager as the source of truth.

edit: I had written this overall comment (including options 3 and 4 here) during the vgo period prior to Go 1.11, and at that point in time option 3 or 4 might have been more reasonable, but now that Go 1.11 has been released, I don't believe that either option 3 or 4 would typically be recommended, and hence the strikethrough.

Additional strategies for v2+ projects

For existing projects that are already on v2.0.0 or higher, the above strategies (such as using go mod -vender) still apply, but there are some additional considerations for v2+ projects that want to support older Go versions.

The primary issue is that v2+ projects that adopt Go 1.11 modules will update their import paths to respect semantic import versioning, with the major version now appearing within the path itself. (In contrast, this issue does not apply to a v0.x.x or v1.x.x Go 1.11 module, because semantic import versioning leaves the import paths unchanged for a v0.x.x or v1.x.x project).

For example, an older Go version such as Go 1.8 does not understand how to properly interpret a /v2 in an import statement that uses the new semantic import versioning conventions. In particular, upon seeing code containing import "example.com/foo/v2/bar", the Go 1.8 tooling would interpret the /v2/ as being part of the file hierarchy.

There are currently three primary solutions for supporting older Go versions in a project on v2 or higher:

1. Go versions starting at 1.9.7+ and 1.10.3+ have been updated to know how to properly consume a v2+ Go 1.11 module, including how to automatically properly interpret a /v2 or higher that appears in a path. This means if your project is willing to restrict support to Go 1.9.7+, 1.10.3+, and 1.11+, then there is nothing else special required for a v2+ module. (Additional details: pre-existing code that directly or indirectly imports a v2+ module does not need to be modified if using Go 1.9.7+, 1.10.3+, and 1.11+. Those Go versions have been updated to make things "just work", without requiring old code to use /v2 or similar in import paths. It is only when code opts-in to modules that /v2 and similar are required in import paths).

2. Alternatively, another option is to make a v2+ Go 1.11 module compatible with the original go get behavior (e.g., to support Go 1.8) by moving or copying the project code into a /v2 subdirectory. For example, if Go 1.8 tooling sees import "example.com/foo/bar/v2", it will interpret v2 as part of the file hierarchy and then successfully find the expected files starting at bar/v2.

   • This approach can make sense for some authors, but some authors will prefer not to copy or move the code into a v2 subdirectory for a transitional period.

3. If neither approach 1. or 2. are appropriate for your v2+ project, then waiting and migrating to modules later is an entirely reasonable choice. As time goes by, Go 1.9.7 and 1.10.3 will have been out for a longer and longer period of time, which means you might be more comfortable making Go 1.9.7 and 1.10.3 required at some future date, including some projects might make the transition when Go 1.12 is released approximately 6 months after Go 1.11.

As additional experience develops with versioned modules, additional solutions might arise from the community.

If you are willing to restrict support to Go 1.9.7+, 1.10.3+, and 1.11+, the additional mechanics to release a v2+ module are as follows (using v2.0.0 as the example):

• Update the go.mod file to include a /v2 at the end of the module path.
• Update import paths to include /v2.
• Tag the release with v2.0.0.
• To avoid confusion with this approach, consider putting the v2.*.* commits on a separate v2 branch.
  • If you instead you have been previously releasing on master and would prefer to tag v2.0.0 on master, then consider creating a v1 branch for any future v1 bug fixes.

If instead you would like to support even older Go releases than Go 1.9.7+, 1.10.3+, the additional mechanics to release a v2+ module are:

• Create a v2 directory and place a new go.mod file in that directory. The module path must end with /v2.
• Copy or move the code into that v2 directory. This could be automated.
• Update import paths to include /v2.
• Tag the release with v2.0.0.

Finally, even if you chose not to opt-in to Go 1.11 modules now, please start using semantic versioning to tag your releases if not already doing so. This will help the overall ecosystem evolve more smoothly.

@marwan-at-work had a similar question: “Should package authors with existing v2+ versions bump the major version again when adding a module definition?”

I'm planning to discuss with @rsc when he gets back and will post some advice here.

Our conclusion was that “changing the import path” is definitely a breaking change, and authors with existing v2+ versions at non-v2+ import paths will need to change the import path when adding a module definition, so they should bump the major version again when they add the module definition.

I think that upgrading a major version only to add support for Go modules is quite severe. Major versions should be bumped for (major) changes in the library interface, not for an administrative detail like adding Go modules (which has nothing to do with the functionality of the code).

As upgrading to v3 is not desirable for me, and v1 has already been phased-out (and will not support Go modules), I'm not sure how to proceed with adding support for Go modules. I was at one point convinced that the +incompatible would come to my rescue, but it seems to be picking an older version automatically, instead of picking the latest release, why? It also seems to prefer the v0.0.0-HASH notation over v2.3.4+incompatible.

What is the advice you can give when bumping a major version is not an option, regarding releases, contents of go.mod module name and go.mod import name/version?

@tdewolff from the client perspective, you need to go get github.com/some/project@v2.3.4 explicitly to get the v2.3.4+incompatible version. Otherwise, go get would fetch latest 1.x.x version instead.

@tdewolff I advice to ignore go.mod until you plan to release next major version. It's a pain, but I don't think there's any other workaround (so you don't break import paths).

I was at one point convinced that the +incompatible would come to my rescue, but it seems to be picking an older version automatically, instead of picking the latest release, why? It also seems to prefer the v0.0.0-HASH notation over v2.3.4+incompatible.

The support for +incompatible versions is only for versions that predate module support, since the existing import path may have come to implicitly mean that version. For actual modules, the module path must be unique for each major version (per semantic import versioning).

Our conclusion was that “changing the import path” is definitely a breaking change, and authors with existing v2+ versions at non-v2+ import paths will need to change the import path when adding a module definition, so they should bump the major version again when they add the module definition.

@bcmills
This conclusion, IMHO, needs to be documented somewhere since I have seen that question come up multiple times. see; gofrs/uuid#61

update; that conclusion is available at; https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher

@komuw I'll one up this and I'll argue that it is dangerous for the solution to such a serious issue be a "by convention" best practice that you must have prior knowledge of and hidden behind a wall of documentation. I will argue that:

1. This is actually a bug. I think this sort of import inconsistency should not be caused by a user deciding that module support is a minor version increment feature. Per how semantic versioning is generally understood (and if the comments in this thread and the growing frequency of this problem are to be believed), opting into a package manager is barely a minor revision change much less a major breaking change of the API. I think there is some flaw here that needs to be addressed.
2. At the very least if indeed we go with the opinion that module support is a major breaking change, I think there should be no potential workflow there the user is able to make this mistake. If we are going to tell people that opting into modules is a major version increment, and provide no options to the contrary, then I believe the tool should go all the way with that and ensure the version of the package is incremented appropriately.

Utilizing some "version recommendation" features proposed in this issue I think would at least solve 2 in sort of a "two birds one stone" way. #26420

Still sticking to my guns about 1 though ;)

@CameronAckermanSEL, it is possible to add module support without incrementing the major version. However, the migration process is a bit more delicate: you end up needing to leave forwarding packages at the old import path to ensure that users in GOPATH mode and in module mode can both continue to resolve updates to the module.

Since the package import paths in a v2+ module will change anyway, you may as well change them to a new major version at the same time.

So incrementing the major version is, strictly speaking, just a recommendation. It's easier for users and maintainers to reason about, but it's not the only possible approach.

@bcmills thank you very much for correcting my assumptions, I appreciate this observation because it helps me better understand some of the limitations behind a core prevention measure in the tool.

That said, after chewing on it this morning, it seems to me that the damage caused by flubbing it is so severe that perhaps it should just increment the major revision as a low hanging fruit fix. But at the same time, there's probably a better solution for this issue. I do think this is an extremely scary behavior and it makes me nervous to consume packages because I'm not sure if they've opted in correctly.

"if you are adopting modules for the first time for a pre-existing repository or set of packages that have already been tagged v2.0.0 or higher before adopting modules, then the recommended best practice is to increment the major version when first adopting modules." - wiki

So I think we're okay.