package main

import (
	"fmt"
	"go/importer"
	"go/token"
	"go/types"
	"os"
	"path/filepath"
	"runtime"

	"golang.org/x/tools/go/gcexportdata"
)

func main() {
	val := filepath.Join(runtime.GOROOT(), "pkg/mod")
	os.Setenv("GOMODCACHE", val)
	fmt.Println("GOMODCACHE:", val)

	var fset = token.NewFileSet()
	var imp types.Importer
	if true {
		imp = importer.ForCompiler(fset, "source", nil)
	} else {
		packages := make(map[string]*types.Package)
		imp = gcexportdata.NewImporter(fset, packages)
	}

	_, err := imp.Import("go/types")
	fmt.Println("Import result:", err)

	_, err = imp.Import("golang.org/x/tools/go/gcexportdata")
	fmt.Println("Import result:", err)
}

got expected result:

GOMODCACHE: /Users/xushiwei/go/pkg/mod
Import result: <nil>
Import result: <nil>

But it ran slowly (not in this example, but in a real package).

CC @golang/tools-team

As of https://go.dev/cl/310515, gcexportdata.NewImporter will be deprecated, IIUC for the reasons present in this CL. Can you use x/tools/go/packages instead?

CC @rsc @matloob

As of https://go.dev/cl/310515, gcexportdata.NewImporter will be deprecated, IIUC for the reasons present in this CL. Can you use x/tools/go/packages instead?

CC @rsc @matloob

I need a types.Importer, but x/tools/go/packages hasn't. And, packages.Load is slow and problematic. If I call packages.Load multiple times, it will create multiple instances of same depended packages. And therefore fails to checking identical types by go/types.

@xushiwei depending on your use-case, it is probably not too hard to implement an importer by memoizing the response of packages.Load. If you don't ask for source/syntax, this should load from export data. If that is still too slow it is worth profiling to understand why.

I'd be interested in hearing how this goes.

@xushiwei depending on your use-case, it is probably not too hard to implement an importer by memoizing the response of packages.Load. If you don't ask for source/syntax, this should load from export data. If that is still too slow it is worth profiling to understand why.

I'd be interested in hearing how this goes.

I had written an importer by packages.Load. There is a lot of trouble here.

For example, suppose package E depends A and B, and package F depends A and C.

When I loaded E, it created an A package instance (called it A1) and B. And when I loaded F, it also create an A package instance (called it A2) and C.

The big trouble is: E depends A1, F depends A2, but types.Identical(A1.Foo, A2.Foo) is NOT true.

A normal importer doesn't have this problem. When I load E, it created an A package instance (called it A1) and B. And when I loaded F, it reused A1 and only create a C package instance.

I had written an importer by packages.Load. There is a lot of trouble here

Right, sorry I didn't really think through my suggestion.

Next suggestion: use go/packages only to get the correct exportfile (using NeedsExportFile), and use gcexportdata.Read. Example here. Then you can use memoized imports... and I think it should work.

FWIW, after https://go.dev/cl/310515 it looks gcexportdata.Import should actually work correctly (despite being deprecated), so you could also test at that CL. As the commit message indicates using go/packages is preferable, but if you are only loading one package at a time they are probably equivalent.

@findleyr Thx. I will try your suggestion.

@findleyr I try

pkgs, err := packages.Load(&packages.Config{Dir: dir, Mode: packages.NeedExportsFile}, "fmt")

and it returns ENOENT.

But I use go list -export directly:

type listExport struct {
	Export string `json:"Export"`
}

func findExport(dir, pkgPath string) (expfile string) {
	var ret listExport
	if data, err := golistExport(dir, pkgPath); err == nil {
		json.Unmarshal(data, &ret)
	}
	return ret.Export
}

func golistExport(dir, pkgPath string) (ret []byte, err error) {
	var stdout, stderr bytes.Buffer
	cmd := exec.Command("go", "list", "-json", "-export", pkgPath)
	cmd.Stdout = &stdout
	cmd.Stderr = &stderr
	cmd.Dir = dir
	err = cmd.Run()
	if err == nil {
		ret = stdout.Bytes()
	}
	return
}

It works well.

@xushiwei

go/types/resolver.go collectObjects() line:259

fileDir := dir(check.fset.Position(file.Name.Pos()).Filename)
check.importPackage(d.spec.Path, path, fileDir)

Has the same problem as you.Maybe not fixed yet.
If you use

check.Files(files)   //types.check 

to check source code. It will cause the same error