cmd/go: support module-local install/run of tool dependencies #27653
Comments
myitcv
added
NeedsDecision
bcmills
changed the title
|
(CC @rsc) |
|
The |
|
Another option might be some sort of |
|
Have you tried retool? I am successfully using that for all my tool installs. |
@gregwebs thanks for the link, yes that had come across my radar before. I think we should avoid dependencies beyond the standard Go distribution where possible, particularly for, what some might consider, something as fundamental as this. |
|
I think this is especially an issue not to fix with third party tools. How would you ensure that the third party tool to install tools can be easily run by all developers? It creates a circular dependency that's entirely avoided if this mechanism is available in the Go toolchain. |
|
Since you asked, the way I ensure it can be run by all developers is with I agree with the sentiment though that a package manager should solve this issue. However,
There's another desire here that is somewhat related: running any command, not just a tool in the environment that contains tool installs, the packages, and other supported environment modifications. In the |
Agreed. This is very much the approach being followed by https://github.com/rogpeppe/gohack, too. Depending on the lessons learned,
I think my main concern is simply that the guidance for every developer globally then becomes:
at which point you've "broken" one of the key benefits of Go. Not least because this combined guidance will not, by definition, be part of the official docs. Hence why I'm keen that we get the functionality "interfaces" right here. It's also worth bearing in mind that modules are still officially experimental in 1.11, so there is still time. |
|
As something of an experiment in this space, would appreciate feedback etc on https://github.com/myitcv/gobin |
|
👏 In regards to the |
A bit more background on what we're trying to experiment with and why given here: https://github.com/myitcv/gobin/wiki/FAQ
(The |
|
Sorry, but I think this is mission creep. It's beyond the scope of the go command to virtualize your whole development environment. The go command is concerned with building code. If you want to manage versions of helper tools and all that, you probably need a more complex build system around it. |
|
To elaborate on my last comment, another important consideration is that we want Go generally to fit in with the surrounding host system, not try to be its own self-contained bubble. I've worked on self-contained bubbles before, most notably Plan 9, and they can be tremendously elegant and productive. But they are also fundamentally isolating. They make it much harder to use things outside the bubble. If you want to use tools for development, I want you to use the tools that work best for you, no matter what language they are written in. The protobuf compiler is two binaries, one in C++ and one in Go. You may have other important tools written in C, or Python, or Rust, or any number of other languages. The common denominator is that they are all programs you can exec. I don't want to privilege tools written in Go in some way that creates a bit of a bubble that makes it harder to use tools written in other languages. And that's exactly what I see happening here. The general problem of exec-able tool versioning is not language-specific. It would be best not to give it a language-specific solution. In contrast, the problem of versioning of importable Go packages is absolutely language-specific, so it's reasonable for the go command to have a language-specific solution. There's probably an interesting system that could be built around versioned bin directories and the like. But that should be a general system, not tied to Go. |
|
I like gobin because it seems to solves 95% of the problem which is tools written in go without requiring other tooling (outside the go bubble!). I like the concept of being language agnostic even better. The So the solution advocated by @rsc may be satisfieable by |
Thanks for the comprehensive response; I understand and agree with the decision to not do this in the We will continue with the |
|
There isn't an official solution to simulate the old I ended up copying & installing from $GOPATH/pkg/mod I think I'll use @myitcv gobin experiment. |
|
@rsc What's the plan with the behaviour of go get once GOPATH disappears? Will go get error if you run that command in a directory without a go.mod? |
|
Note that this issue is closed. If you are running into a concrete problem that requires a response, please open a new one. |
|
@bcmills What's the intended behavior in Go 1.12 of go get outside a module but with modules enabled? Do binaries get installed anywhere? If so, where in the absence of GOPATH? |
|
In Go 1.12 the |
|
Erm, I might be a little late and I stumbled upon this issue myself. I am able to run a specific tool with specific version defined in go.mod without "external" tool. All following commands expect I have to do it this way because of Makefile, you can split the commands to your fit An example with go:generate
So basically I wanted a And then use it as prefix to the corresponding go file. With go run you can still use arguments (as here with the second "run") |
golint changed its import path [1], and that along with the advent of modules caused fallout [2, 3] that broke the `go get -u` installation in our makefile/CI build. The tools.go idiom is the currently favored approach for versioning development tools with the module system [4, 5], in a way that `go mod tidy` won't churn them from `go.mod` and the `+build` constraint keeps them out of actual build products. The tools still need to be `go install`ed, within a module `go get -u` is not the thing to do anymore because it upgrades transitive deps of a tool which may change the module's build. It takes like hours of reading discussions to triangulate on these moving targets... [5, 6, 7, 8] jfc how much of life have I spent following the fashion evolution of Go dependency management [1]: golang/lint@c363707 [2]: golang/go#30455 [3]: golang/go#30831 [4]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module [5]: golang/go#25922 [6]: golang/go#27653 [7]: golang/go#27643 [8]: golang/go#30515
In GH-82 and GH-88, two workarounds have been implemented in order to prevent the "pollution" of the project Go module file due to development tools and dependencies when installed through `go get`. The workaround to install modules/packages outside of the project root directory (preventing the Go toolchain to pick up the `$GOMOD` environment variable initialized with the path to the projects Go module file) works, but might result in problems due to already installed executables with different versions. The general problem of tool dependencies a a long-time known issue/weak point of the current Go toolchain and is a highly rated change request from the Go community [1,2]. The official Go GitHub repository wiki provides a section on "How can I track tool dependencies for a module?" [3] that describes a workaround that tracks tool dependencies through the Go module logic via a `tools.go` file with a dedicated `tools` build tag to prevent these modules to be included in production binary artifact builds. This approach works fine for non-main packages, but for CLI tools that are only implemented in the `main` package can not be imported in such a file. In order to tackle this problem, a user from the community implemented `gobin` [4], "an experimental, module-aware command to install/run main packages". It allows to install or run main-package commands without "polluting" the Go module file by default. It downloads modules in version-aware mode into a binary cache path within the system's cache directory (`os.UserCacheDir()` [5]). It can be used to query for the path of the executable for a given module/package to simplify the usage from within Mage. It prevents problems due to already installed global binaries in `$GOPATH`/`$GOBIN` by using a cache directory instead. This keeps the system clean and ensures the correct version of a module executable is already used. `gobin` is still in an early development state, but has already received a lot of positive feedback and is used in many projects. There are also many members of the core Go team that are contributing to the project and the chance is high that it will influence the official future Go toolchain implementation or might be partially ported. To finally manage the tool dependency problem for snowsaw, `gobin` has been integrated into the Mage build toolchain. [1]: golang/go#25922 [2]: golang/go#27653 [3]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module [4]: https://github.com/myitcv/gobin [5]: https://golang.org/pkg/os/#UserCacheDir Relates to GH-82 Relates to GH-88 GH-90
In GH-82 and GH-88, two workarounds have been implemented in order to prevent the "pollution" of the project Go module file due to development tools and dependencies when installed through `go get`. The workaround to install modules/packages outside of the project root directory (preventing the Go toolchain to pick up the `$GOMOD` environment variable initialized with the path to the projects Go module file) works, but might result in problems due to already installed executables with different versions. The general problem of tool dependencies a a long-time known issue/weak point of the current Go toolchain and is a highly rated change request from the Go community [1,2]. The official Go GitHub repository wiki provides a section on "How can I track tool dependencies for a module?" [3] that describes a workaround that tracks tool dependencies through the Go module logic via a `tools.go` file with a dedicated `tools` build tag to prevent these modules to be included in production binary artifact builds. This approach works fine for non-main packages, but for CLI tools that are only implemented in the `main` package can not be imported in such a file. In order to tackle this problem, a user from the community implemented `gobin` [4], "an experimental, module-aware command to install/run main packages". It allows to install or run main-package commands without "polluting" the Go module file by default. It downloads modules in version-aware mode into a binary cache path within the system's cache directory (`os.UserCacheDir()` [5]). It can be used to query for the path of the executable for a given module/package to simplify the usage from within Mage. It prevents problems due to already installed global binaries in `$GOPATH`/`$GOBIN` by using a cache directory instead. This keeps the system clean and ensures the correct version of a module executable is already used. `gobin` is still in an early development state, but has already received a lot of positive feedback and is used in many projects. There are also many members of the core Go team that are contributing to the project and the chance is high that it will influence the official future Go toolchain implementation or might be partially ported. Also see gobin's FAQ page in the repository wiki [6] for more details. To finally manage the tool dependency problem for snowsaw, `gobin` has been integrated into the Mage build toolchain. [1]: golang/go#25922 [2]: golang/go#27653 [3]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module [4]: https://github.com/myitcv/gobin [5]: https://golang.org/pkg/os/#UserCacheDir [6]: https://github.com/myitcv/gobin/wiki/FAQ Relates to GH-82 Relates to GH-88 Resolves GH-90
>>> Go Executable Installation When installing a Go executable from within a Go module [1] directory using the `go install` command [2], it is installed into the Go executable search path that is defined through the `GOBIN` environment variable [3] and can also be shown and modified using the `go env` command [4]. Even though the executable gets installed globally, the `go.mod` file [5] will be updated to include the installed packages since this is the default behavior of the `go get` command [6] running in "module mode" [7]. Next to this problem, the installed executable will also overwrite any executable of the same module/package that was installed already, but maybe from a different version. Therefore only one version of a executable can be installed at a time which makes it impossible to work on different projects that use the same tool but with different versions. >>>> History & Future The local installation of executables built from Go modules/packages has always been a somewhat controversial point which unfortunately, partly for historical reasons, does not offer an optimal and user-friendly solution up to now. The `go` command [8] is a fantastic toolchain that provides many great features one would expect to be provided out-of-the-box from a modern and well designed programming language without the requirement to use a third-party solution: from compiling code, running unit/integration/benchmark tests, quality and error analysis, debugging utilities and many more. Unfortunately the way the `go install` command [2] of Go versions less or equal to 1.15 handles the installation of an Go module/package executable is still not optimal. The general problem of tool dependencies is a long-time known issue/weak point of the current Go toolchain and is a highly rated change request from the Go community with discussions like golang/go#30515 [9], golang/go#25922 [10] and golang/go#27653 [11] to improve this essential feature, but they've been around for quite a long time without a solution that works without introducing breaking changes and most users and the Go team agree on. Luckily, this topic was finally picked up for the next upcoming Go release version 1.16 [12] and golang/go#40276 [13] introduces a way to install executables in module mode outside a module. The release note preview also already includes details about this change [14] and how installation of executables from Go modules will be handled in the future. >>>> The Workaround Beside the great news and anticipation about an official solution for the problem the usage of a workaround is almost inevitable until Go 1.16 is finally released. The official Go wiki [15] provides a section on "How can I track tool dependencies for a module?" [16] that describes a workaround that tracks tool dependencies. It allows to use the Go module logic by using a file like `tools.go` with a dedicated `tools` build tag that prevents the included module dependencies to be picked up included for normal executable builds. This approach works fine for non-main packages, but CLI tools that are only implemented in the `main` package can not be imported in such a file. In order to tackle this problem, a user from the community created "gobin" [17], an experimental, module-aware command to install/run main packages. It allows to install or run main-package commands without "polluting" the `go.mod` file by default. It downloads modules in version-aware mode into a binary cache path within the systems cache directory [18]. It prevents problems due to already globally installed executables by placing each version in its own directory. The decision to use a cache directory instead of sub-directories within the `GOBIN` path keeps the system clean. "gobin" is still in an early development state, but has already received a lot of positive feedback and is used in many projects. There are also members of the core Go team that have contributed to the project and the chance is high that the changes for Go 1.16 were influenced or partially ported from it. It is currently the best workaround to... 1. prevent the Go toolchain to pick up the `GOMOD` environment variable [4] (see [`go env GOMOD`][4]) that is initialized automatically with the path to the `go.mod` file in the current working directory. 2. install module/package executables globally without "polluting" the `go.mod` file. 3. install module/package executables globally without overriding already installed executables of different versions. See gobin's FAQ page [19] in the repository wiki for more details about the project. >>>> The Go Module Caster To allow to manage the tool dependency problem, wand now uses "gobin" through a new caster [20] `go.mod` file and allows to... 1. install `gobin` itself into `GOBIN` (`go env GOBIN` [4]). 2. cast any spell incantation [21] of kind `KindGoModule` [22] by installing the executable globally into the dedicated `gobin` cache. [1]: https://golang.org/ref/mod [10]: golang/go#25922 [11]: golang/go#27653 [12]: https://github.com/golang/go/milestone/145 [13]: golang/go#40276 [14]: https://tip.golang.org/doc/go1.16#modules [15]: https://github.com/golang/go/wiki [16]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module [17]: https://github.com/myitcv/gobin [18]: https://golang.org/pkg/os/#UserCacheDir [19]: https://github.com/myitcv/gobin/wiki/FAQ [2]: https://golang.org/cmd/go#hdr-Compile_and_install_packages_and_dependencies [20]: https://pkg.go.dev/github.com/svengreb/wand/pkg/cast/gobin#Caster [21]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#Incantation [22]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#KindGoModule [3]: https://golang.org/cmd/go/#hdr-Environment_variables [4]: https://golang.org/cmd/go/#hdr-Print_Go_environment_information [5]: https://golang.org/ref/mod#go-mod-file [6]: https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them [7]: https://golang.org/ref/mod#mod-commands [8]: https://golang.org/cmd/go [9]: golang/go#30515 GH-22
>>> Go Executable Installation When installing a Go executable from within a Go module [1] directory using the `go install` command [2], it is installed into the Go executable search path that is defined through the `GOBIN` environment variable [3] and can also be shown and modified using the `go env` command [4]. Even though the executable gets installed globally, the `go.mod` file [5] will be updated to include the installed packages since this is the default behavior of the `go get` command [6] running in "module mode" [7]. Next to this problem, the installed executable will also overwrite any executable of the same module/package that was installed already, but maybe from a different version. Therefore only one version of a executable can be installed at a time which makes it impossible to work on different projects that use the same tool but with different versions. >>>> History & Future The local installation of executables built from Go modules/packages has always been a somewhat controversial point which unfortunately, partly for historical reasons, does not offer an optimal and user-friendly solution up to now. The `go` command [8] is a fantastic toolchain that provides many great features one would expect to be provided out-of-the-box from a modern and well designed programming language without the requirement to use a third-party solution: from compiling code, running unit/integration/benchmark tests, quality and error analysis, debugging utilities and many more. Unfortunately the way the `go install` command [2] of Go versions less or equal to 1.15 handles the installation of an Go module/package executable is still not optimal. The general problem of tool dependencies is a long-time known issue/weak point of the current Go toolchain and is a highly rated change request from the Go community with discussions like golang/go#30515 [9], golang/go#25922 [10] and golang/go#27653 [11] to improve this essential feature, but they've been around for quite a long time without a solution that works without introducing breaking changes and most users and the Go team agree on. Luckily, this topic was finally picked up for the next upcoming Go release version 1.16 [12] and golang/go#40276 [13] introduces a way to install executables in module mode outside a module. The release note preview also already includes details about this change [14] and how installation of executables from Go modules will be handled in the future. >>>> The Workaround Beside the great news and anticipation about an official solution for the problem the usage of a workaround is almost inevitable until Go 1.16 is finally released. The official Go wiki [15] provides a section on "How can I track tool dependencies for a module?" [16] that describes a workaround that tracks tool dependencies. It allows to use the Go module logic by using a file like `tools.go` with a dedicated `tools` build tag that prevents the included module dependencies to be picked up included for normal executable builds. This approach works fine for non-main packages, but CLI tools that are only implemented in the `main` package can not be imported in such a file. In order to tackle this problem, a user from the community created "gobin" [17], an experimental, module-aware command to install/run main packages. It allows to install or run main-package commands without "polluting" the `go.mod` file by default. It downloads modules in version-aware mode into a binary cache path within the systems cache directory [18]. It prevents problems due to already globally installed executables by placing each version in its own directory. The decision to use a cache directory instead of sub-directories within the `GOBIN` path keeps the system clean. "gobin" is still in an early development state, but has already received a lot of positive feedback and is used in many projects. There are also members of the core Go team that have contributed to the project and the chance is high that the changes for Go 1.16 were influenced or partially ported from it. It is currently the best workaround to... 1. prevent the Go toolchain to pick up the `GOMOD` environment variable [4] (see [`go env GOMOD`][4]) that is initialized automatically with the path to the `go.mod` file in the current working directory. 2. install module/package executables globally without "polluting" the `go.mod` file. 3. install module/package executables globally without overriding already installed executables of different versions. See gobin's FAQ page [19] in the repository wiki for more details about the project. >>>> The Go Module Caster To allow to manage the tool dependency problem, wand now uses "gobin" through a new caster [20] `go.mod` file and allows to... 1. install `gobin` itself into `GOBIN` (`go env GOBIN` [4]). 2. cast any spell incantation [21] of kind `KindGoModule` [22] by installing the executable globally into the dedicated `gobin` cache. [1]: https://golang.org/ref/mod [10]: golang/go#25922 [11]: golang/go#27653 [12]: https://github.com/golang/go/milestone/145 [13]: golang/go#40276 [14]: https://tip.golang.org/doc/go1.16#modules [15]: https://github.com/golang/go/wiki [16]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module [17]: https://github.com/myitcv/gobin [18]: https://golang.org/pkg/os/#UserCacheDir [19]: https://github.com/myitcv/gobin/wiki/FAQ [2]: https://golang.org/cmd/go#hdr-Compile_and_install_packages_and_dependencies [20]: https://pkg.go.dev/github.com/svengreb/wand/pkg/cast/gobin#Caster [21]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#Incantation [22]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#KindGoModule [3]: https://golang.org/cmd/go/#hdr-Environment_variables [4]: https://golang.org/cmd/go/#hdr-Print_Go_environment_information [5]: https://golang.org/ref/mod#go-mod-file [6]: https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them [7]: https://golang.org/ref/mod#mod-commands [8]: https://golang.org/cmd/go [9]: golang/go#30515 Closes GH-22
What version of Go are you using (
go version)?What did you do?
Tied to install and run a tool dependency in a number of my modules (more detail below)
What did you expect to see?
A nice easy way to (install and) run a tool dependency.
What did you see instead?
I needed to set
GOBINand updatePATHfor each module.Further detail (on What did you do?)
This issue builds on top of #25922 so if that changes shape in any significant way it may void what follows.
Per #27643 (comment), I think we need to make the workflow around using a module's tool dependencies easier. Let me try to explain by covering my "workflow".
The tools I use on a day-to-day basis fall into two categories:
go.modDealing with these in reverse order.
Category 2: I think it's clear that with Go 1.11, there is a "gap" here and that this is covered by #24250. Per the detail in that discussion, there are open questions on how to handle multiple versions, where the installed binaries should be put etc, but it all falls under that issue.
Category 1: by far the largest category of tools for me, made up largely of code generators that I use with
go generateand the like. I absolutely want these to be version controlled. And I don't want to be using (via myPATH) a "global" install of such a tool, even if the version just happens to match at that point in time. But bothgo getandgo installcurrently (i.e. Go 1.11) have a target of$GOPATH/bin(ignoring multi-element `GOPATH values for now).Hence the workflow I have effectively adopted, building on #25922, is to create a module-local install target:
As covered in #27643, one of the disconnects in Go 1.11 is that a
go getorgo installin a module context modifies the "local"go.modbut installs "globally". This is, as @bcmills put it, "weird". But is to my mind a gap in Go 1.11, just as there not being a "global" tool install is a gap (i.e. #24250).Potential solutions
Just listing these here as a starting point:
go runis a potential alternative to the "local" install here (and a very attractive one to my mind), but we need to find a way to address cmd/go: go run pkg is significantly slower than running built binary #25416..bin/directory, alongside ago.mod, is the target forgo getandgo install(ofmainpackages) for "local" installs? But this wouldn't obviate the need for everyone to update theirPATHand indeed.gitignorethe.bindirectory for every module they work on.cc @bcmills @rogpeppe @mvdan
The text was updated successfully, but these errors were encountered: