Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: suppress warning messages about symlinks #35941

Closed
cespare opened this issue Dec 3, 2019 · 15 comments
Closed

cmd/go: suppress warning messages about symlinks #35941

cespare opened this issue Dec 3, 2019 · 15 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@cespare
Copy link
Contributor

cespare commented Dec 3, 2019

I'm converting a large source tree which used vendoring in GOPATH world into a Go module. An issue I've encountered is that go mod tidy and go mod why print some warnings about symlinks.

These look like:

$ go mod tidy
warning: ignoring symlink /path/to/some/symlink/dir
...

Where a warning line is printed for every symlinked directory in the repo.

These warnings make these 'go mod' commands unpleasant to use (you have to scroll through all the junk to see the relevant bits, if any, at the bottom).

None of the Go code is in symlinked directories. The symlinks are for other projects in other languages. For example, one use of symlinks is as a cheap way to share certain JS/CSS assets between multiple sub-projects.

Would it be possible to simply delete these warnings? Alternatively, could we remove the warning if the symlinked directory doesn't contain any Go code?

I tested this behavior in both Go 1.13.4 and tip (bf3ee57).

/cc @bcmills @jayconrod for cmd/go, and @rsc who originally wrote these warning messages as far as I can tell.

@jayconrod
Copy link
Contributor

It looks like this warning was introduced in CL 46425, long before modules. It's emitted any time a pattern is expanded (all or anything with ...) and a symlink to a directory is encountered. go mod tidy is doing that internally.

I don't think these warnings should be deleted entirely, but making them less spammy would be good. Not sure how to do that though. Checking for .go files in a symlinked directory is probably not enough, since the directory itself may not have any .go files, but it may have subdirectories that do. I think Kubernetes' vendor/k8s.io directory works like that (with symbolic links to staging). Traversing subdirectories after a symbolic link could get us into a loop though.

@jayconrod jayconrod added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 3, 2019
@jayconrod jayconrod added this to the Backlog milestone Dec 3, 2019
@cespare
Copy link
Contributor Author

cespare commented Dec 3, 2019

@jayconrod yeah, I've run into this in the past when doing things like go list ./....

However, I think the spam is more of a problem with modules:

  • go list is used a lot less in the day-to-day workflow in the GOPATH world than go mod commands are used in the modules world.
  • With go list ./... or similar, it's more understandable that you're asking the Go tool to please go discover all the packages. With go mod tidy it's less clear.

What about suppressing the warnings for go mod subcommands only? (And keeping them for go list?)

@jayconrod
Copy link
Contributor

Suppressing these warnings for go mod subcommands seems reasonable to me. @bcmills WDYT?

@bcmills
Copy link
Contributor

bcmills commented Dec 4, 2019

If they're not appropriate for go mod, they're probably not appropriate for go list either. (And the converse: if they're useful for go list, then they're probably also useful for go mod.)

Given that folks can already explicitly prune out subdirectories using go.mod files, I'm curious about the directory structure that leads to this sort of situation. Are the non-Go symlinks in sibling directories to the Go source files, or in subdirectories of Go package directories?

As an alternative, would it make sense to do the go.mod pruning check before we do the symlink-pruning check?

@cespare
Copy link
Contributor Author

cespare commented Dec 5, 2019

If they're not appropriate for go mod, they're probably not appropriate for go list either. (And the converse: if they're useful for go list, then they're probably also useful for go mod.)

I explained above why I think it's reasonable to treat these differently, but if we have to treat them the same, then I'd err on the side of no spam.

Given that folks can already explicitly prune out subdirectories using go.mod files, I'm curious about the directory structure that leads to this sort of situation.

We have a large monorepo with projects in several languages. Before it was all a single GOPATH with vendored source. I'm converting it to be a single module instead.

Adding empty go.mod files to a bunch of project directories just to suppress this warning is probably a nonstarter.

Are the non-Go symlinks in sibling directories to the Go source files, or in subdirectories of Go package directories?

Currently, only sibling (or "cousin" or "cousin Nth removed") directories.

I didn't mind ignoring these warnings (or 2>/dev/null) when running go list in the GOPATH world. It just doesn't seem reasonable that I have to continue doing so with commands like go mod tidy, which is part of my normal workflow.

@agnivade
Copy link
Contributor

agnivade commented Dec 5, 2019

Just to add a small point, we have a situation where a Go directory is linked with a symlink. (I know .. I know ..). So removing these warning would certainly help.

@bcmills
Copy link
Contributor

bcmills commented Dec 5, 2019

Currently, only sibling (or "cousin" or "cousin Nth removed") directories.

So, could you move the go.mod file to the root of the Go source tree, instead of the repo root? (Or does it need access to testdata in sibling directories?)

@cespare
Copy link
Contributor Author

cespare commented Dec 5, 2019

So, could you move the go.mod file to the root of the Go source tree, instead of the repo root?

There is no single Go source tree. Everything's mixed. What I meant was only that no Go package directory or any descendent thereof contains symlinks.

├─go.mod
├─go.sum
├─go─project─1/
├─go─project─2/
├─non─go─project─1/
│ └─symlink
└─sub/
  ├─go─project─3/
  ├─non─go─project─2/
  │ └─symlink
  └─symlink

@sebd71
Copy link

sebd71 commented Jan 13, 2020

As workaround you can create an empty go.mod file in each directory you want to ignore.

touch no-go-project-1/go.mod
touch sub/no-go-project-2/go.mod

But this is not a perfect workaround because you ignore sub/symlink with this method.

See go wiki modules https://github.com/golang/go/wiki/Modules#can-an-additional-gomod-exclude-unnecessary-content-do-modules-have-the-equivalent-of-a-gitignore-file

@cespare
Copy link
Contributor Author

cespare commented Jan 13, 2020

@sebd71 as I wrote above:

Adding empty go.mod files to a bunch of project directories just to suppress this warning is probably a nonstarter.

It's simply not reasonable for us to add lots of go.mod files to non-Go projects, especially in an ongoing way (adding and deleting them as we add and delete symlinks).

@sebd71
Copy link

sebd71 commented Jan 13, 2020

I'm agree with you and that's why I said that it's not a perfect workaround.
I'm also facing the same problem.
A good solution would be to add in root go.mod "exclude mydir/*" statement, but unfortunately it seems not supported for now.

@cespare
Copy link
Contributor Author

cespare commented Jun 2, 2020

@bcmills @jayconrod would you accept a PR for this once the tree opens? We still haven't migrated our code to modules and this tooling wart is definitely a sticking point.

@jayconrod
Copy link
Contributor

Discussed with @matloob, @bcmills offline. It seems like the best thing to do here is only print this warning when there's an explicit argument that has a ..., and the go command does not follow a symbolic link to a directory when expanding that pattern. The warning should not be printed for all, nor should it be printed for go mod tidy.

@cespare Does that work for you?

I'd be happy to review a CL for this. We can review any time, but it won't be merged until the tree opens for 1.16.

@cespare
Copy link
Contributor Author

cespare commented Jun 4, 2020

@cespare Does that work for you?

Absolutely. Thanks!

rayjcwu added a commit to liftoffio/go that referenced this issue Apr 20, 2021
Go commands show a warning message any time a pattern is expanded and a
symlink to a directory is encountered. For monorepo with non Go projects
using symlinks underneath, the output of go commands could be spammed by
this warning.

This commit includes the behavior change to only print this warning when
there's a pattern containing ... .

Fixes golang#35941
@gopherbot
Copy link

Change https://golang.org/cl/311890 mentions this issue: cmd/go: show warnings about symlinks only for patterns containing ...

@bcmills bcmills modified the milestones: Backlog, Go1.17 Apr 27, 2021
@golang golang locked and limited conversation to collaborators Apr 27, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants