/cc @griesemer

@griesemer can you look at this issue? I have a very simple repo case. Basically, if a package in a go module depends on a package that has a vendor directory, conf.Check returns an error. Maybe the packages aren't returned from parser.ParseDir?

@dimes. Ok, will look at this next week.

Thanks. I created a really simple example. Here is a package with a vendor directory: https://github.com/dimes/testtesttest

You can repro this by creating a new Go module:

1. go mod init some/package
2. go get github.com/dimes/testtesttest
3. Create some file that imports the package:

package mypkg;

import "github.com/dimes/testtesttest"

type SomeInterface interface {
	SomeFunction() testtesttest.FooBar
}

4. In the example in the first issue post, set packageName to some/package (the input to go mod init).

I cannot reproduce this issue.

Here's what I have:

• $GOPATH/src/github.com/dimes contains your testtesttest package incl. the vendored directory.
• $GOPATH/src/mypkg contains mypkg.go
• I used the following package containing your code from your original comment: x.go (in $HOME/tmp).
• I'm running it via go run x.go in $HOME/tmp.

The x.go package traces various things, including imports. I'm seeing:

$ go run x.go
imported.Dir = /Users/gri/src/mypkg
package mypkg
    - /Users/gri/src/mypkg/mypkg.go
importing "github.com/dimes/testtesttest" (dir = /Users/gri/src/mypkg, mode = 0) => package testtesttest ("github.com/dimes/testtesttest") (err = <nil>)

You are using import.ForCompiler(fileSet, "source", nil). The "source" mode requires that all source files are present, recursively.
Are you sure all the source .go files are present? (Looking at your error message, do all those files exist in your file system?)

More generally: The parser.ParseDir function is ancient and from a time before we had go/build. Now that go/build provides a build.Package, you should use the files from that Package, parse those individually (via parser.ParseFile) and then provide them to types.Config.Check (which is already built to only accept a list of files. The ast.Packages returned by ParseDir do not have complete information; for instance, they ignore +build tags (though this doesn't matter in this case).

Please double-check your situation and reply. Thanks.

Thanks @griesemer. I will try using build.Package now. One difference between our two setups is that you have your $GOPATH set and my testtesttest package inside. I did not have a $GOPATH set and was using go mod to both fetch testtesttest and run the code. Sorry for omitting that.

@griesemer I just tried using build.Package in the following way:

	var files []*ast.File
	imported, err := build.Default.Import(packageName, ".", 0)
	if err != nil {
		return nil, errors.Wrapf(err, "Error importing package %s", packageName)
	}

	for _, goFile := range imported.GoFiles {
		fileName := filepath.Join(imported.Dir, goFile)
		fmt.Println("Parsing", fileName)
		file, err := parser.ParseFile(fileSet, fileName, nil, parser.AllErrors)
		if err != nil {
			return nil, errors.Wrapf(err, "Error parsing go file %s", fileName)
		}
		files = append(files, file)
	}

	info := &types.Info{
		Defs: make(map[*ast.Ident]types.Object),
	}

	conf := types.Config{
		Importer: importer.ForCompiler(fileSet, "source", nil),
	}

	_, err = conf.Check(packageName, fileSet, files, info) // err is non-nil

The error is the same. I think there is a weird interaction here when using Go modules.

@griesemer sorry for the spam, but I believe I figured out the root cause of this issue. Inside of Checker.collectObjects, the imports are extracted from the files and then imported here: https://golang.org/src/go/types/resolver.go#L256

The issue seems to be that:

check.importPackage(s.Path.Pos(), path, fileDir)

Is using fileDir as the directory to do the import from. Internally, check.importPackage uses the provided Importer to do the import. When the $GOPATH is set, this is fine. However, when using go mod, the importer looks for a go.mod file in the srcDir given to Import call. The fileDir of the dependency does not have a go.mod file, so it falls back to using the $GOPATH. It does not find the package. If I change importPackage call in resolver.go to be:

check.importPackage(s.Path.Pos(), path, ".") // go.mod located at "./go.mod" 

then it works. Not sure what the general solution to this is.

@dimes Thanks for this analysis, that is very helpful. Looping in @bcmills who has been working on module support for additional input.

(The fileDir argument provided to importPackage was added in the past to deal with vendor directories since vendored packages have to be lookup up relative to the directory of the package that is doing the import. This has all changed with modules. It may be that the importer implementations may have to be updated.)

CC @matloob @ianthehat

Hi,

Unless I'm misunderstanding this, this is an issue with go/build rather than go/types.

The go/build package has some very rudimentary support for modules, but we do not intend to expand on that support. It was only added as a backstop to help support a small set of programs because a replacement that could handle modules didn't yet exist. Take a look at Russ's change in f851253 for more context on that support.

The golang.org/x/tools/go/packages package should now be used load and type-check packages in tools, instead of go/build.

I'm wondering if it would be okay to add a deprecation notice to go/build stating its shortcomings and redirecting users to golang.org/x/tools/go/packages.

@matloob Whatever is not working "as expected" in go/build in module-aware mode should be documented and/or deprecated as necessary. Otherwise how should one determine if something is working correctly?

@dimes Please report back if you cannot get your code to work with x/tools/go/packages. I have retitled this issue according to @matloob 's feedback. I'm also leaving this open for now. It may be "fixed" by documenting the shortcomings of go/build. There's an effort underway to update a lot of the relevant documentation for module-aware mode. Apologies for the inconvenience.

In the meantime until we can figure out whether go/build as a whole can be deprecated, it seems safe to document that it's not supported for modules use cases.

I'll send a CL to do that.

@dimes x/tools/go/packages should support your use case. Please let me know if it doesn't so we can try to fix it.

Change https://golang.org/cl/183848 mentions this issue: go/build: document that it's not supported for modules

Ok, great. Thanks for the help. Using the packages package simplified things a lot:

	config := &packages.Config{
		Mode: packages.LoadSyntax,
	}
	pkgs, err := packages.Load(config, packageName)
	if err != nil {
		return nil, errors.Wrapf(err, "Error loading package %s", packageName)
	}

	for _, pkg := range pkgs {
		for identifier, definition := range pkg.TypesInfo.Defs {
			// ...
		}
	}