x/tools/go/packages: vendor is not searched in GOPATH mode #30289

vektah · 2019-02-18T07:12:08Z

What version of Go are you using (`go version`)?

$ go version
go version go1.11.5 linux/amd64

Does this issue reproduce with the latest release?

Reproducable on 1.12-rc1

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/vektah/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/vektah/go"
GOPROXY=""
GORACE=""
GOROOT="/home/vektah/local/go"
GOTMPDIR=""
GOTOOLDIR="/home/vektah/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build863238177=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Trying to load a package using x/tools/go/packages doesn't seem to search vendor directories.

For your convenience this code can be fetched from https://github.com/vektah/packages-vendor-test

packages_test.go

package main_test

import (
	"github.com/external/tool"
	"golang.org/x/tools/go/packages"
	"testing"
)

func TestPackageLoading(t *testing.T) {
	// we can reference the vendored package just fine
	if tool.Foo() != true {
		t.Error("vendor imports are broken?")
	}

	// but if we try to load the package via packages
	pkgs, err := packages.Load(nil, "github.com/external/tool")
	if err != nil {
		t.Error(err.Error())
	}

	// we get an error!
	if pkgs[0].Errors != nil {
		t.Error(pkgs[0].Errors)
	}

	// and a blank package name
	if pkgs[0].Name != "tool" {
		t.Error("the package name should be set")
	}
}

vendor/github.com/external/tool/tool.go

package tool

func Foo() bool {
	return true
}

What did you expect to see?

PASS
ok      github.com/vektah/packages-vendor-test  0.001s

What did you see instead?

--- FAIL: TestPackageLoading (0.01s)
    packages_test.go:24: [-: cannot find package "github.com/external/tool" in any of:
                /home/vektah/local/go/src/github.com/external/tool (from $GOROOT)
                /home/vektah/go/src/github.com/external/tool (from $GOPATH)]
    packages_test.go:29: the package name should be set
FAIL
exit status 1
FAIL    github.com/vektah/packages-vendor-test  0.008s

This is hurting go modules adoption in http://github.com/99designs/gqlgen, Ideally I should be able to switch to packages to get modules support without losing support for vendor.

The text was updated successfully, but these errors were encountered:

mvdan · 2019-02-18T08:08:21Z

I presume you're running the tool with GO111MODULE=on?

Also, see https://golang.org/cmd/go/#hdr-Maintaining_module_requirements; go build will not use vendor/ by default when ran in module mode, unlesss -mod=vendor, is given.

Also see #30240, which I believe would make this better and automatic in 1.13.

vektah · 2019-02-18T08:30:53Z

I presume you're running the tool with GO111MODULE=on?

No, I'm using GO111MODULE=auto from inside the gopath. The same failure happens when running GO111MODULE=off.

Note that in the test I am importing the vendored package directly to prove the go toolchain is happy to resolve this path, its only x/packages that cant find it.

mvdan · 2019-02-18T08:37:50Z

Interesting - I'm not sure why this would happen, as go/packages uses go list underneath. If you've vendored package foo/bar via vendor/foo/bar, what happens if you run go list -compiled foo/bar?

/cc @matloob @ianthehat

vektah · 2019-02-18T08:48:42Z

go list -compiled github.com/external/tool
can't load package: package github.com/external/tool: cannot find package "github.com/external/tool" in any of:
        /usr/local/go/src/github.com/external/tool (from $GOROOT)
        /home/adam/go/src/github.com/external/tool (from $GOPATH)

interestingly this is ok

go list -compiled github.com/vektah/packages-vendor-test/vendor/github.com/external/tool
github.com/vektah/packages-vendor-test/vendor/github.com/external/tool

which is consistent with go build

go build github.com/external/tool
can't load package: package github.com/external/tool: cannot find package "github.com/external/tool" in any of:
        /usr/local/go/src/github.com/external/tool (from $GOROOT)
        /home/adam/go/src/github.com/external/tool (from $GOPATH)

go build github.com/vektah/packages-vendor-test/vendor/github.com/external/tool
(OK)

mvdan · 2019-02-18T09:37:18Z

If I add a main.go file importing github.com/external/tool, go list -deps does the right thing:

$ go list -deps -f {{.Dir}}
/home/mvdan/go/src/github.com/vektah/packages-vendor-test/vendor/github.com/external/tool
/home/mvdan/tip/src/internal/cpu
/home/mvdan/tip/src/unsafe
/home/mvdan/tip/src/internal/bytealg
/home/mvdan/tip/src/runtime/internal/atomic
/home/mvdan/tip/src/runtime/internal/sys
/home/mvdan/tip/src/runtime/internal/math
/home/mvdan/tip/src/runtime
/home/mvdan/go/src/github.com/vektah/packages-vendor-test

So I really don't know where the bug is. Seems to me like go list works fine.

mvdan · 2019-02-18T09:39:54Z

Ah, I think I'm finally catching up - you can't do packages.Load(cfg, "github.com/external/tool") because go list github.com/external/tool doesn't work.

I think go/packages, like go list -deps, will look in the vendor folder to find dependencies, but not the direct packages you've asked it to load.

From https://golang.org/cmd/go/#hdr-Vendor_Directories:

Go 1.6 includes support for using local copies of external dependencies to satisfy imports of those dependencies, often referred to as vendoring.

So it seems to me like this is by design; vendor is only meant to satisfy dependencies.

Perhaps this will all work better in module mode with the new vendor folder; I'm not sure. /cc @bcmills

vektah · 2019-02-18T10:09:00Z

I think that dates back to a bunch of fixes to prevent tools that didn't know about vendoring from accidentally running across all of the vendored code. Hiding it from go list makes sense for the same reason.

I don't think this logic should be applied to x/packages, its a new module conceived well after vendoring was added, and its purpose is very different. x/loader used to check vendor, and x/packages checks the module path for dependencies, vendor search seems like a large omision here.

At the very least it should have a flag that mirrors ImportMode IgnoreVendor in build.

matloob · 2019-02-26T19:48:18Z

Hello @vektah, this is working as intended.

go/packages defers to the build system's (in this case, go build's and go list's) understanding of patterns you supply to it. So in your case, go/packages understands packages as go list does.

When run in GOPATH mode, go list's output is independent of the directory it's run in (with one exception below). And the way go list is specified in GOPATH mode, there's no way to supply which package you're "importing" a package from. Because we can't specify the importer of a package, go list has no way of knowing which vendor directories to consult to resolve a package.

The exception is that you can specify relative and absolute directory paths to go list. And as you noticed you can provide the qualified package paths pointing into the vendor directories. But it doesn't sound like either of those would solve your problem.

If you're trying to find a vendored package by its import path, your best bet is to do a deps query of an importer and look through those deps to find the package that you need.

vektah · 2019-02-26T22:57:28Z

this is working as intended.

Ok, that's a little disappointing. What is the intended use case for packages? Maybe I've misunderstood this to be a replacement for x/loader, because x/loader was not going to receive go modules updates?

I think not having a viable successor to X/loader is a big contributer to #24661

If you're trying to find a vendored package by its import path, your best bet is to do a deps query of an importer and look through those deps to find the package that you need.

I can't really do that, the code doesn't exist yet (this is for a codegen tool). I guess I could generate a new file in the right dir with a a bunch of _ imports, but that seems like a lot of work.

I might end up having to reproduce the is vendoring check and call go build directly for package resolution, before handing off to packages.

matloob · 2019-02-27T16:07:51Z

We do plan for go/packages to be a viable successor to loader, but it can't have all the features loader had because we want it both to be more correct and to be compatible with different build systems.

I'd like to know more about your use case to see if there's some other way for your tool to get the information it needs through the go/packages interface. I'm especially interested in how the vendored package name is provided to go/packages (by the user, by a codegen comment, is it created by the code generator?), and what the vendored package's relation is to the generated code.

If you'd like, I'd be able to meet via VC to discuss.

vektah · 2019-03-03T23:39:02Z

Its a code generator that takes some graphql schema and config that maps it to go types to bridge between the two type systems.

There are a heap of builtin graphql types like Int and String that are implemented in gqlgen, so when searching for those types it needs to find them in vendor. There are also often things like UUIDs and api types that are implemented in external libraries and referenced like this.

Its complicated, but if that isn't really clear I'm happy to jump on a call.

matloob · 2019-03-04T20:37:03Z

Are those types vendored into gqlgen itself? If so, can you use the "fully-qualified" package path to load the package? If not, where is it vendored?

vektah · 2019-03-25T22:53:28Z

Are those types vendored into gqlgen itself? If so, can you use the "fully-qualified" package path to load the package? If not, where is it vendored?

Those ones are, but they arent all defined by gqlgen, gqlgen is just adding its default implementations to config. End users can also refer directly to other external dependencies, eg well known protobuffer types and these full import paths shouldnt be leaking out.

For now we are now converting the standard import path from users to these full paths by calling build.Import before passing it off to x/packages.

matloob · 2019-03-26T18:21:53Z

Okay, I think this is a case of "working as intended; unfortunate". go list doesn't provide a way to look up that information, so we can't support it in go/packages. I think the workaround is reasonable for now, until everyone moves over to modules...

x/tools/go/pacakges does not search inside vendor. Work around by using build.Import first to find the package in vendor. See golang/go#30289

x/tools/go/packages does not search inside vendor. Work around by using build.Import first to find the package in vendor. See golang/go#30289

seankhliao · 2022-12-26T13:18:09Z

Closing as working as intended then

gopherbot added this to the Unreleased milestone Feb 18, 2019

mvdan added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 18, 2019

mvdan changed the title ~~x/tools/go/packages: vendor is not searched~~ x/tools/go/packages: vendor is not searched in GOPATH mode Feb 18, 2019

vektah mentioned this issue Mar 4, 2019

Workaround for using packages with vendored code 99designs/gqlgen#570

Merged

agnivade added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 22, 2019

nkovacs mentioned this issue Jul 15, 2019

counterfeiter does not check vendor directory when generating using import path maxbrunsfeld/counterfeiter#75

Closed

nkovacs added a commit to nkovacs/counterfeiter that referenced this issue Jul 15, 2019

Fix importing of vendored packages

94813b2

x/tools/go/pacakges does not search inside vendor. Work around by using build.Import first to find the package in vendor. See golang/go#30289

nkovacs added a commit to nkovacs/counterfeiter that referenced this issue Jul 15, 2019

Fix importing of vendored packages

4b91f4b

x/tools/go/packages does not search inside vendor. Work around by using build.Import first to find the package in vendor. See golang/go#30289

nkovacs mentioned this issue Jul 15, 2019

Fix importing of vendored packages maxbrunsfeld/counterfeiter#136

Merged

gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Sep 12, 2019

seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale Dec 26, 2022

golang locked and limited conversation to collaborators Dec 26, 2023

gopherbot added the FrozenDueToAge label Dec 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/go/packages: vendor is not searched in GOPATH mode #30289

x/tools/go/packages: vendor is not searched in GOPATH mode #30289

vektah commented Feb 18, 2019

mvdan commented Feb 18, 2019

vektah commented Feb 18, 2019 •

edited

mvdan commented Feb 18, 2019

vektah commented Feb 18, 2019

mvdan commented Feb 18, 2019

mvdan commented Feb 18, 2019

vektah commented Feb 18, 2019 •

edited

matloob commented Feb 26, 2019

vektah commented Feb 26, 2019

matloob commented Feb 27, 2019

vektah commented Mar 3, 2019

matloob commented Mar 4, 2019

vektah commented Mar 25, 2019

matloob commented Mar 26, 2019

seankhliao commented Dec 26, 2022

x/tools/go/packages: vendor is not searched in GOPATH mode #30289

x/tools/go/packages: vendor is not searched in GOPATH mode #30289

Comments

vektah commented Feb 18, 2019

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

mvdan commented Feb 18, 2019

vektah commented Feb 18, 2019 • edited

mvdan commented Feb 18, 2019

vektah commented Feb 18, 2019

mvdan commented Feb 18, 2019

mvdan commented Feb 18, 2019

vektah commented Feb 18, 2019 • edited

matloob commented Feb 26, 2019

vektah commented Feb 26, 2019

matloob commented Feb 27, 2019

vektah commented Mar 3, 2019

matloob commented Mar 4, 2019

vektah commented Mar 25, 2019

matloob commented Mar 26, 2019

seankhliao commented Dec 26, 2022

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

vektah commented Feb 18, 2019 •

edited

vektah commented Feb 18, 2019 •

edited