This is behaving as expected.

Please see https://github.com/golang/go/wiki/Modules#why-does-go-mod-tidy-record-indirect-and-test-dependencies-in-my-gomod

I'm not sure I agree. That link talks about build tags and all active modules, but what happens here is that clearly inactive modules are pulled in.

go mod why -m $module_path should be able to give you an import path to each module. If it can't, then there is a bug somewhere.

There is more discussion of this particular issue #27920 in a recent golang-nuts thread (also started by @alicebob, I think), where that thread might have answered at least some of the questions raised in this issue:
https://groups.google.com/forum/#!topic/golang-nuts/Cl--P02fCjw

@alicebob, in terms of your comment above about "clearly inactive modules are pulled in", the modules documentation has a particular definition for the phrase "active modules" (from https://tip.golang.org/cmd/go/#hdr-List_packages_or_modules, with emphasis added):

The main module is the module containing the current directory. The active modules are the main module and its dependencies.

In any event, setting aside for the moment if you think the go tooling should behave in a different manner, do you see anywhere that go mod why -m $module_path does not give you an explanation of why a dependency is present?

@thepudds thanks for your replies. Yeah, I posted that to get more answers.

go mod why does explain all modules in my example repo. I did manage to get a module in a go.mod which wasn't used; I think it was a sequence of running tidy, and then using go get to upgrade a module. But I didn't manage to reproduce that yet.

So I guess things work indeed mostly "as intended" :(

I see now that vendor does not include all the modules only included in tests by vendored modules. So what's in go.mod is not simply what's added in the vendor folder, and it's smarter than that.

Great! I missed that completely. It is indeed described in go help mod vendor. My bad.

To answer my own initial question: yes, go.mod is long, but since that's not the list of modules which get vendored it doesn't matter.

Thanks @thepudds and @myitcv!

Here is a case which is not clear to me and I think is related to this issue:

Whenever a dependency is added to the code I can just run go get or go build and it will automatically be added to go.mod with all the required dependencies.
The counterpart of this behavior is whenever I remove some code importing dependencies and those dependencies are not imported anywhere else in my module, they should get removed.

This is how I think the majority of projects work and worked before using tools like dep and glide.

Compared to this go mod tidy adds all transitive dependencies (even ones not required to build my project) to the go.mod file.

Now, I understand the reasons (I think) why it works like that, but as I said before, I don't think (the majority of) Go projects work or want to work like that. Let me give an example:

It was mentioned in the google groups thread (and it is my understanding as well), that in a CI all dependencies in the go.mod files will be downloaded. This can be a huge amount of dependencies making CI builds very slow. Imagine a library providing some sort of kubernetes integration (in a separate package of the module). Even if I don't use that, the kubernetes repo will be downloaded (since it's in the project's go.mod and it should be), which is friendly 2GB in size.

I think in most of the cases we need a solution that keeps the go.mod an absolute minimum with the dependencies that are directly or indirectly imported during go build or go test.

I understand that there can be cases when one would like to run tests for all the dependencies, but I also think that's probably not the majority of the projects. (Sticking to the example: I don't want to run the test suite of kubernetes)

In any case, it would be nice to have an option for the above, or at least an option to limit "tidying" to removal of unused dependencies, leaving new dependencies to go build.

I may not have all the facts right, but after a few hours of reading and playing around this is what I ended up with.

whenever I remove some code importing dependencies and those dependencies are not imported anywhere else in my module, they should get removed.

Note that “not imported anywhere else” is an expensive property to detect. That's why go mod tidy is a separate operation.

Compared to this go mod tidy adds all transitive dependencies (even ones not required to build my project) to the go.mod file.

go mod tidy adds all dependencies needed to satisfy the (transitive) imports of the packages in your module, including their tests. go mod why should be able to explain each of those dependencies. If you find an example for which that is not the case, please file a separate issue with steps to reproduce it.

I think in most of the cases we need a solution that keeps the go.mod an absolute minimum with the dependencies that are directly or indirectly imported during go build or go test.

In the general case, the go.mod file is the minimal set of dependencies required to make go test all reproducible.

in a CI all dependencies in the go.mod files will be downloaded. This can be a huge amount of dependencies making CI builds very slow.

According to the documentation:

With no arguments, download applies to all dependencies of the main module.

It's not obvious to me whether “dependencies of the main module” should refer to package dependencies or module dependencies (that's #26902), but in this case it doesn't matter, because there is an alternative that downloads only the dependencies of the main module:

go list -test -deps ./...

which tells the go command to resolve the transitive non-test dependencies of all packages and tests in the main module (and necessarily downloads them to the cache in the process).

Thanks for clarification. The only missing thing for me then is the remove functionality of tidy. Since in most of the cases I don't want to run go test all, go mod tidy is probably not for me.

To be honest go mod feels a bit fragile for this use case: one accidental go mod tidy and my go.mod is doomed forever. (Obviously I can try to revert, given I notice it in time, or remove it and start over, remove indirect deps, but I think non of those are optimal solutions)

I wonder though if I'm right about my assumption, that a majority of Go projects doesn't want to use go test all, thus go mod tidy simply won't be used, despite it having the removal of unnecessary dependencies.

@sagikazarmark This issue was closed, and as far as I am following things here, the original problem reporter seemed to indicate they were satisfied with the discussion here:

Great! I missed that completely. It is indeed described in go help mod vendor. My bad.

To answer my own initial question: yes, go.mod is long, but since that's not the list of modules which get vendored it doesn't matter.

Thanks

Reading over again what you wrote here, it might be a bit different than the original problem report?

Would it make sense to open a new issue for what you are reporting, or do you think your open questions/issues are already covered by another set of issue(s), or is your view that this issue here should be re-opened?

Sorry, just trying to see what is pending, vs. not.

@thepudds I think I'm going to open a new issue (unless I found another one). Although I understand the explanation given in this issue, after experimenting with modules for a while I still think the behavior above is undesirable.