Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: confusion about modules not kicking on in GOPATH/src by default #26394

Closed
alecthomas opened this issue Jul 15, 2018 · 14 comments
Closed
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@alecthomas
Copy link

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version devel +22e17d0ac7 Sun Jul 15 09:19:41 2018 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

darwin amd64

What did you do?

go mod -init on a project that imports a dependency that vendors and imports a mutual shared dependency. ie. A -> C, A -> B -> C

What did you expect to see?

The mutual shared dependency should be resolved to the same import, but is not.

I can understand why it doesn't support this, but it makes piecemeal migration difficult. Packages would have to be migrated in the order of their dependencies.

What did you see instead?

cmd/internal/auth/auth.go:35:42: cannot use creds (type *"importedproject/vendor/github.com/aws/aws-sdk-go/aws/credentials".Credentials) as type *"github.com/aws/aws-sdk-go/aws/credentials".Credentials in argument to connectors.NewAWSConnector
@kardianos
Copy link
Contributor

Is it possible for you to share an example repo that duplicates this? I'm not sure how it pulled one dep from a module and another dep from a vendor folder.

@alecthomas
Copy link
Author

Not trivially, no, this is in a private repo.

Repository A vendors repository B. Both repository A and B have github.com/aws/aws-sdk-go vendored. Running go mod -init in repository A worked fine, but compilation failed with the above.

@alecthomas
Copy link
Author

I should also mention that vendoring is currently managed with dep.

@oiooj oiooj added the modules label Jul 16, 2018
@oiooj oiooj added this to the Go1.11 milestone Jul 16, 2018
@bcmills bcmills changed the title Mutual vendored shared dependencies don't resolve to the same package cmd/go: mutual vendored shared dependencies don't resolve to the same package Jul 16, 2018
@bcmills
Copy link
Contributor

bcmills commented Jul 16, 2018

In module mode, the go command intentionally ignores /vendor directories.

Repository A vendors repository B. Both repository A and B have github.com/aws/aws-sdk-go vendored.

Which vendored copy do you expect the build to use?

If both A and B vendor the same pristine copy of github.com/aws/aws-sdk-go, then you should be able to go get github.com/aws/aws-sdk-go@$VERSION in both modules and have everything work as expected, independent of the /vendor directory.

On the other hand, if the vendored copy has changes that are intentionally not compatible with a tagged release of aws-sdk-go, then either it should have a distinct import path (e.g. github.com/alecthomas/aws-sdk-go) or a replace directive (replace github.com/aws/aws-sdk-go => github.com/alecthomas/aws-sdk-go v1.15.0) in both modules.

@bcmills
Copy link
Contributor

bcmills commented Jul 16, 2018

I should also mention that vendoring is currently managed with dep.

If the vendored copy of aws-sdk-go is a pristine upstream version specified in your Gopkg.lock file, does go mod -init correctly add a require directive for that version?

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 16, 2018
@alecthomas
Copy link
Author

Which vendored copy do to expect the build to use?

It's the same pristine version, so I don't care which one it uses.

If both A and B vendor the same pristine copy of github.com/aws/aws-sdk-go, then you should be able to go get github.com/aws/aws-sdk-go@$VERSION in both modules and have everything work as expected, independent of the /vendor directory.

Only one of the two repositories has been "go modularised", repository A.

@alecthomas
Copy link
Author

One question, should I be able to delete the vendor directory at this stage of development, or is that still a work in progress?

@bcmills
Copy link
Contributor

bcmills commented Jul 17, 2018

If you don't need reproducible builds without module support, you should be able to delete the vendor directory. (In module mode, you should be able to build without it.)

That's a pretty big “if”, but it's your repo. 🙂

@rsc
Copy link
Contributor

rsc commented Jul 17, 2018

The report is a bit hard to follow since it is so abstract, but this error looks wrong to me:

```
cmd/internal/auth/auth.go:35:42: cannot use creds (type 
*"importedproject/vendor/github.com/aws/aws-sdk-go/aws/credentials".Credentials) as type 
*"github.com/aws/aws-sdk-go/aws/credentials".Credentials in argument to connectors.NewAWSConnector
```

When using go modules, vendor directories should be ignored entirely. It should not be possible to end up with a package "importedproject/vendor/github.com/aws/aws-sdk-go/aws/credentals" in the build at all. It sounds like "importedproject/somethig" imports "github.com/aws/aws-sdk-go/aws/credentials" and that resolved to the vendored copy instead of the real one chosen for the overall build. If so, that's a mistake. We just need to isolate the test case showing why it happens.

A little more information would help. Can you post your go.mod (with "importedproject" or whatever is fine)? Can you tell if importedproject imports "github.com/aws/aws-sdk-go/aws/credentials"? Does it import it using that exact path? Or maybe does it import it using "importedproject/vendor/github.com/aws/aws-sdk-go/aws/credentals? (That should be impossible but we've already established there's a bug somewhere.)

Thanks.

@rsc rsc changed the title cmd/go: mutual vendored shared dependencies don't resolve to the same package cmd/go: module build somehow using vendor directory in dependency Jul 17, 2018
@alecthomas
Copy link
Author

I've replicated this in some minimal public repositories:

$ go-tip get github.com/alecthomas/test0
$ cd $GOPATH/src/github.com/alecthomas/test0
$ go-tip install ./...
$ go-tip mod -init
go: creating new go.mod: module github.com/alecthomas/test0
go: copying requirements from Gopkg.lock
$ go-tip install ./...
$ rm -rf vendor
$ go-tip install ./...
main.go:6:2: cannot find package "github.com/alecthomas/test1" in any of:
	/Users/aat/bin/go.tip/src/github.com/alecthomas/test1 (from $GOROOT)
	/Users/aat/.go/src/github.com/alecthomas/test1 (from $GOPATH)
$ go-tip get ./...
# github.com/alecthomas/test0
./main.go:12:7: cannot use arn1 (type "github.com/alecthomas/test1/vendor/github.com/aws/aws-sdk-go/aws/arn".ARN) as type "github.com/aws/aws-sdk-go/aws/arn".ARN in assignment
./main.go:13:7: cannot use arn0 (type "github.com/aws/aws-sdk-go/aws/arn".ARN) as type "github.com/alecthomas/test1/vendor/github.com/aws/aws-sdk-go/aws/arn".ARN in assignment

Apologies for not having a replicable example before.

As I mentioned initially, I'm not sure this specific combination of actions is expected to work at all, or is even reasonable to expect to work.

@bcmills
Copy link
Contributor

bcmills commented Jul 18, 2018

It is expected to work with GO111MODULE=on, but since the code is in $GOPATH/src, it should fall back to a non-module build (i.e., using /vendor) in auto mode.

@alecthomas
Copy link
Author

Aha! Setting GO111MODULE=on does indeed work. Sorry for the noise :(

@rsc
Copy link
Contributor

rsc commented Jul 18, 2018

I'm really sorry but the problem here is that modules are not enabled for your commands. I will take this as a sign that we need to make the non-module usage inside GOPATH/src clearer. Maybe even a warning print, as much as I hate those. Perhaps go mod -init should have failed too.

$ export GOPATH=/tmp/gp1
$ go get github.com/alecthomas/test0
$ cd $GOPATH/src/github.com/alecthomas/test0
$ go install ./...
$ go mod -init
go: creating new go.mod: module github.com/alecthomas/test0
go: copying requirements from Gopkg.lock
$ go install ./...
$ rm -rf vendor
$ go install ./...
main.go:6:2: cannot find package "github.com/alecthomas/test1" in any of:
	/Users/rsc/go/src/github.com/alecthomas/test1 (from $GOROOT)
	/tmp/gp1/src/github.com/alecthomas/test1 (from $GOPATH)
main.go:4:2: cannot find package "github.com/aws/aws-sdk-go/aws/arn" in any of:
	/Users/rsc/go/src/github.com/aws/aws-sdk-go/aws/arn (from $GOROOT)
	/tmp/gp1/src/github.com/aws/aws-sdk-go/aws/arn (from $GOPATH)
$ GO111MODULE=on go install ./...
go: finding github.com/aws/aws-sdk-go v1.14.27
go: finding github.com/alecthomas/test1 v0.0.0-20180717001517-0b8cedefc96b
go: downloading github.com/aws/aws-sdk-go v1.14.27
go: downloading github.com/alecthomas/test1 v0.0.0-20180717001517-0b8cedefc96b
$ 

@rsc rsc changed the title cmd/go: module build somehow using vendor directory in dependency cmd/go: confusion about modules not kicking on in GOPATH/src by default Jul 18, 2018
@alecthomas
Copy link
Author

Ah, of course. That makes complete sense now.

@golang golang locked and limited conversation to collaborators Jul 18, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

6 participants