Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: behavior of go list ... is not documented in module mode #37227

Open
perillo opened this issue Feb 14, 2020 · 17 comments
Open

cmd/go: behavior of go list ... is not documented in module mode #37227

perillo opened this issue Feb 14, 2020 · 17 comments
Labels
Documentation GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@perillo
Copy link
Contributor

perillo commented Feb 14, 2020

go help packages documents the special path all both for gopath and module mode:

"all" expands to all packages found in all the GOPATH trees. For example, 'go list all' lists all the packages on the local system. When using modules, "all" expands to all packages in the main module and their dependencies, including dependencies needed by tests of any of those.

However the ... pattern is documented only for gopath module:

An import path is a pattern if it includes one or more "..." wildcards, each of which can match any string, including the empty string and strings containing slashes. Such a pattern expands to all package directories found in the GOPATH trees with names matching the patterns.

How is the ... pattern supposed to work in module mode?

As a test, I ran go list ... | wc -l inside the github.com/golang/mod (golang.org/x/mod) directory with different versions of the go tool. The directory is outside GOPATH. The results are:

  • go1.11.13: 501
  • go1.12.16: 526
  • go1.13.8: 632
  • go1.14rc1: 634

As a side note, how can I tell go list to list only packages in the main module?

@perillo
Copy link
Contributor Author

perillo commented Feb 14, 2020

Unrelated to this issue, I also noted that with GO111MODULE=off and GOPATH set to a not existing directory:

  • go1.13.8 list ...: 300
  • go1.13.8 list std cmd: 376

This seems to contrast with the documentation saying Such a pattern expands to all package directories found in the GOPATH, since it is matching package directories in GOROOT.

The result is the same with GO111MODULE=on and inside an empty module (where go list all matches no packages).

@dmitshur dmitshur added this to the Backlog milestone Feb 14, 2020
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 14, 2020
@dmitshur
Copy link
Contributor

/cc @jayconrod @bcmills @matloob per owners.

@perillo
Copy link
Contributor Author

perillo commented Feb 14, 2020

I think that go list ... should never match packages outside the main and active modules.

The difference between go list ... and go list std cmd is that the latter also includes the vendor directories.

@bcmills
Copy link
Contributor

bcmills commented Feb 14, 2020

It is true that we have not documented the ... package pattern in module mode. The intent is for it to have the same meaning as in GOPATH mode: every package that can be found in any of the available package roots. (In GOPATH mode, the available package roots are GOROOT/src, and GOPATH/src for each entry in GOPATH. In module mode, the available package roots are GOROOT/src and each active module.)

The increase from Go 1.11 to 1.12 is due to the change from vendor/golang_org/x to internal/x for packages vendored into the standard library (see CL 147443).

The increase from Go 1.12 to 1.13 is due to the inclusion of the cmd subtree plus the addition of crypto/ed25519. (The previous lack of cmd packages was a bug present since Go 1.11: note that the cmd subtree was always included in GOPATH mode.)

The minor changes from Go 1.13 to 1.14 are due to changes in internal packages plus the addition of hash/maphash.

@bcmills
Copy link
Contributor

bcmills commented Feb 14, 2020

The difference between GOPATH mode and module mode in Go 1.13 is due to the requirement that each package have a unique path in module mode: for example, the package reported as cmd/vendor/golang.org/x/sys/unix would be just one of many possible packages known as golang.org/x/sys/unix in GOPATH mode.

Arguably the lack of such a listing entirely in GOPATH mode is a bug, but it is a fairly minor one and at this point almost certainly not worth fixing.

@perillo
Copy link
Contributor Author

perillo commented Feb 14, 2020

There is still something missing, in module mode.

For go1.13, the number of packages in GOROOT is 376 (and ... only matches 300 packages).

Inside the golang.org/x/mod module:

  • go1.13 list all: 170
  • go1.13 list std cmd all: 392
  • go1.13 list std cmd ...: 708

Am I missing something?

@bcmills
Copy link
Contributor

bcmills commented Feb 14, 2020

Per https://tip.golang.org/cmd/go/#hdr-Package_lists_and_patterns:

When using modules, "all" expands to all packages in the main module and their dependencies, including dependencies needed by tests of any of those.

Since no package in the main module can import a package main from cmd, that is a strict subset of ....

@perillo
Copy link
Contributor Author

perillo commented Feb 14, 2020

Is it only me that thinks this is really complex?

@bcmills
Copy link
Contributor

bcmills commented Feb 14, 2020

The ... pattern is just not all that useful. It's an edge-case of a general wildcard. So, yeah: edge cases are complex, but generally avoidable. (The all pattern is usually what you want instead, and that one is well-defined.)

@perillo
Copy link
Contributor Author

perillo commented Feb 14, 2020

It could be make useful if ... matches only the packages in the main module, since there are no other means to get that list.

@myitcv
Copy link
Member

myitcv commented Feb 14, 2020

@perillo can you not do

go list $(go list -m)/...

@bcmills
Copy link
Contributor

bcmills commented Feb 14, 2020

I think we do need a pattern that matches only the packages in the main module, but ... is not that pattern.

(./... is such a pattern, but it only works in the module root.)

@bcmills
Copy link
Contributor

bcmills commented Feb 14, 2020

@myitcv: go list $(go list -m)/... will catch packages in nested modules too.

@bcmills
Copy link
Contributor

bcmills commented Feb 14, 2020

@jayconrod, @matloob, and I were discussing such a pattern this morning, in fact.

(We're thinking maybe mod, since it's a three-letter shortening like std and cmd and less likely than main to be used as the name of someone's local toy module.)

But that really ought to be filed as a separate issue.

@perillo
Copy link
Contributor Author

perillo commented Feb 14, 2020

@bcmills I tested go list $(go list -m)/... from inside the golang.org/x/tools directory and it does not list the golang.org/x/tools/gopls sub module.

@bcmills
Copy link
Contributor

bcmills commented Feb 14, 2020

It would if x/tools depended on gopls, which it (intentionally) does not.

I think cloud.google.com/go has that sort of structure, though.

@myitcv
Copy link
Member

myitcv commented Feb 14, 2020

Ok, one last try 😄

go list -f "{{\$p := .}}{{with \$p.Module}}{{if eq .Path \"$(go list -m)\"}}{{\$p.ImportPath}}{{end}}{{end}}" $(go list -m)/...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants