x/tools/go/packages: overlay does not work properly with external module files #71075

mateusz834 · 2024-12-31T21:53:00Z

Consider:

package main

import (
	"fmt"

	"golang.org/x/tools/go/packages"
)

func main() {
	fmt.Println(run())
}

func run() error {
	cfg := packages.Config{
		Mode: packages.LoadSyntax,
		Dir:  ".",
		Overlay: map[string][]byte{
			"/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0/go/packages/nonExistent.go": []byte(`package packages; func TestFunc() {}`),  // New file
			"/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0/go/packages/doc.go":         []byte(`package packages; func TestFunc2() {}`), // File exists, but overridden.

			// File exists, but overridden, additional import (unicode/utf8).
			"/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0/go/packages/loadmode_string.go": []byte(`// Copyright 2019 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package packages

import (
	"fmt"
	"strings"
	"unicode/utf8"
)

func TestFunc3(r rune) []byte {
	return utf8.AppendRune(nil, r)
}

var modes = [...]struct {
	mode LoadMode
	name string
}{
	{NeedName, "NeedName"},
	{NeedFiles, "NeedFiles"},
	{NeedCompiledGoFiles, "NeedCompiledGoFiles"},
	{NeedImports, "NeedImports"},
	{NeedDeps, "NeedDeps"},
	{NeedExportFile, "NeedExportFile"},
	{NeedTypes, "NeedTypes"},
	{NeedSyntax, "NeedSyntax"},
	{NeedTypesInfo, "NeedTypesInfo"},
	{NeedTypesSizes, "NeedTypesSizes"},
	{NeedForTest, "NeedForTest"},
	{NeedModule, "NeedModule"},
	{NeedEmbedFiles, "NeedEmbedFiles"},
	{NeedEmbedPatterns, "NeedEmbedPatterns"},
}

func (mode LoadMode) String() string {
	if mode == 0 {
		return "LoadMode(0)"
	}
	var out []string
	// named bits
	for _, item := range modes {
		if (mode & item.mode) != 0 {
			mode ^= item.mode
			out = append(out, item.name)
		}
	}
	// unnamed residue
	if mode != 0 {
		if out == nil {
			return fmt.Sprintf("LoadMode(%#x)", int(mode))
		}
		out = append(out, fmt.Sprintf("%#x", int(mode)))
	}
	if len(out) == 1 {
		return out[0]
	}
	return "(" + strings.Join(out, "|") + ")"
}
`),
		},
	}
	pkgs, err := packages.Load(&cfg, "./test")
	if err != nil {
		return err
	}
	packages.PrintErrors(pkgs)
	return nil
}

This program runs packages.Load on a ./test package, with three overlay files on the golang.org/x/tools/go/packages package.

nonExistent.go - File does not exist, so this is a new file with a new func TestFunc.
doc.go - File already exists, it defines a new func TestFunc2.
loadmode_string.go - File already exists, it defines a new func TestFunc3 and imports unicode/utf8 (this package is not imported without overlays).

The ./test package contains following file:

package main

import (
	"golang.org/x/tools/go/packages"
)

func main() {
	packages.TestFunc()
	packages.TestFunc2()
	packages.TestFunc3('a')
}

$ go run .
-: # golang.org/x/tools/go/packages
/tmp/gocommand-708375563/3-loadmode_string.go:10:2: could not import unicode/utf8 (open : no such file or directory)
/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0/go/packages/loadmode_string.go:10:2: could not import unicode/utf8 (no metadata for unicode/utf8)
/tmp/aa/test/test.go:8:11: undefined: packages.TestFunc
<nil>

So:

It is not able to import unicode/utf8 (probably go list does not return it to go/packages).
Newly added files are ignored (nonExistent.go).
Overlay works, but only for files that already exist.

Opened this issue, because i think that this behavior is wrong, it should either work without any error for the example above, or just ignore the overlay completely.

CC @matloob (per https://dev.golang.org/owners) @adonovan

The text was updated successfully, but these errors were encountered:

gabyhelp · 2024-12-31T21:53:24Z

Related Issues

x/tools/go/packages: returns wrong package name when overlay has a new package name #38328 (closed)
cmd/go: go test does not parse -overlay files for test functions #44946 (closed)
x/tools/go/packages: NeedExportFile causes error about invalid go.mod file to be dropped #67638
x/tools/go/packages: passing NeedSyntax to Load() doesn't populate Syntax field unless NeedTypes is passed #35331 (closed)
x/tools/go/packages: Load in NeedName|NeedFiles mode can produce incorrect results for certain modules #31894 (closed)
x/tools/go/loader: tests fail with go 1.5 tip #11902 (closed)
x/tools/go/packages: clarify naming convention of overlay files #67541
x/tools/go/packages: NeedSyntax without NeedFiles/NeedCompiledGoFiles produces empty slice #56677
x/tools/go/loader: broken tests #11920 (closed)

Related Code Changes

cmd/go: support overlaying go.mod files

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

mateusz834 · 2025-01-01T08:57:37Z

This error comes directly from go list, and the nonExistent.go is not returned (as described in the issue).

GOROOT= GOPATH=/home/mateusz/go GO111MODULE= GOPROXY= PWD=. go list -overlay=/tmp/gocommand-417420243/overlay.json -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,Module -compiled=true -test=false -export=true -deps=true -find=false -pgo=off -- .

{
        "Dir": "/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0/go/packages",
        "ImportPath": "golang.org/x/tools/go/packages",
        "Name": "packages",
        "Module": {
                "Path": "golang.org/x/tools",
                "Version": "v0.28.0",
                "Time": "2024-12-04T22:20:41Z",
                "Dir": "/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0",
                "GoMod": "/home/mateusz/go/pkg/mod/cache/download/golang.org/x/tools/@v/v0.28.0.mod",
                "GoVersion": "1.22.0",
                "Sum": "h1:WuB6qZ4RPCQo5aP3WdKZS7i595EdWqWR8vqJTlwTVK8=",
                "GoModSum": "h1:dcIOrVd3mfQKTgrDVQHqCPMWy6lnhfhtX3hLXYVLfRw="
        },
        "DepOnly": true,
        "GoFiles": [
                "doc.go",
                "external.go",
                "golist.go",
                "golist_overlay.go",
                "loadmode_string.go",
                "packages.go",
                "visit.go"
        ],
        "CompiledGoFiles": [
                "doc.go",
                "external.go",
                "golist.go",
                "golist_overlay.go",
                "loadmode_string.go",
                "packages.go",
                "visit.go"
        ],
        "Imports": [
                "bytes",
                "context",
                "encoding/json",
                "errors",
                "fmt",
                "go/ast",
                "go/parser",
                "go/scanner",
                "go/token",
                "go/types",
                "golang.org/x/sync/errgroup",
                "golang.org/x/tools/go/gcexportdata",
                "golang.org/x/tools/internal/gocommand",
                "golang.org/x/tools/internal/packagesinternal",
                "golang.org/x/tools/internal/typesinternal",
                "encoding/json",
                "errors",
                "fmt",
                "go/ast",
                "go/parser",
                "go/scanner",
                "go/token",
                "go/types",
                "golang.org/x/sync/errgroup",
                "golang.org/x/tools/go/gcexportdata",
                "golang.org/x/tools/internal/gocommand",
                "golang.org/x/tools/internal/packagesinternal",
                "golang.org/x/tools/internal/typesinternal",
                "log",
                "os",
                "os/exec",
                "path",
                "path/filepath",
                "reflect",
                "runtime",
                "slices",
                "sort",
                "strconv",
                "strings",
                "sync",
                "sync/atomic",
                "time",
                "unicode"
        ],
        "Error": {
                "ImportStack": null,
                "Pos": "",
                "Err": "# golang.org/x/tools/go/packages\n/tmp/gocommand-417420243/3-loadmode_string.go:10:2: could not import unicode/utf8 (open : no such file or directory)\n"
        }
}

mateusz834 · 2025-01-01T09:06:03Z

Doing the same thing, but with the x/tools module under go.work (so that it does not use the mod cache):

GOROOT= GOPATH=/home/mateusz/go GO111MODULE= GOPROXY= PWD=. go list -overlay=/tmp/gocommand-2002914138/overlay.json -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,Module -compiled=true -test=false -export=true -deps=true -find=false -pgo=off -- .

{
	"Dir": "/tmp/aa/tools/go/packages",
	"ImportPath": "golang.org/x/tools/go/packages",
	"Name": "packages",
	"Export": "/home/mateusz/.cache/go-build/66/66d1428ddfbc9ee5f6e0c00f1827659c07e965a523f71ce88981612ced939ddd-d",
	"Module": {
		"Path": "golang.org/x/tools",
		"Main": true,
		"Dir": "/tmp/aa/tools",
		"GoMod": "/tmp/aa/tools/go.mod",
		"GoVersion": "1.22.0"
	},
	"DepOnly": true,
	"GoFiles": [
		"doc.go",
		"external.go",
		"golist.go",
		"golist_overlay.go",
		"loadmode_string.go",
		"nonExistent.go",
		"packages.go",
		"visit.go"
	],
	"CompiledGoFiles": [
		"doc.go",
		"external.go",
		"golist.go",
		"golist_overlay.go",
		"loadmode_string.go",
		"nonExistent.go",
		"packages.go",
		"visit.go"
	],
	"Imports": [
		"bytes",
		"context",
		"encoding/json",
		"errors",
		"fmt",
		"go/ast",
		"go/parser",
		"go/scanner",
		"go/token",
		"go/types",
		"golang.org/x/sync/errgroup",
		"golang.org/x/tools/go/gcexportdata",
		"golang.org/x/tools/internal/gocommand",
		"golang.org/x/tools/internal/packagesinternal",
		"golang.org/x/tools/internal/typesinternal",
		"log",
		"os",
		"os/exec",
		"path",
		"path/filepath",
		"reflect",
		"runtime",
		"slices",
		"sort",
		"strconv",
		"strings",
		"sync",
		"sync/atomic",
		"time",
		"unicode",
		"unicode/utf8"
	]
}

matloob · 2025-01-10T22:02:03Z

Hi, I'd like to understand your use case better, but in general this is not supported.

We expect the module cache to be immutable (and we write all the files and directories in the cache readonly), and I think it would be impractical to change that expectation.

One thing we could do is to detect if files were overlaid in the module cache and explicitly reject that at the beginning. We could also do the same, with, for example the build cache.

mateusz834 · 2025-01-11T07:11:35Z

Hi, I'd like to understand your use case better, but in general this is not supported.

I don't have any particular use case, i just mistakenly overridden the module cache files (in the overlay) and spotted this behavior. I am fine if it didn't worked at all. The current behavior and the error message open : no such file or directory (no file name) made me think that this is a bug.

I am not entirely sure what did you mean by "and explicitly reject that at the beginning"? Load (go/packages) returning an error or (as my suggestion) ignoring the overlay files for such cases?

matloob · 2025-01-13T15:43:01Z

I meant to return an error. I'm thinking of things from the go command perspective: go/packages invokes go list to get the information it needs. I'm proposing that the go command returns an error, which is in turn is returned by go/packages.

mateusz834 · 2025-01-13T17:19:59Z

I meant to return an error. I'm thinking of things from the go command perspective: go/packages invokes go list to get the information it needs. I'm proposing that the go command returns an error, which is in turn is returned by go/packages.

Directly by Load or in Package.Error? I think it should be in Package.Error since otherwise it might break LSP (people can accidentally edit module cache files).

matloob · 2025-01-13T20:04:16Z

cc @findleyr

Oh, that's a good point. I was thinking directly by Load, but you're right that that would cause problems. It would be more complicated to figure out how to set the error on the package from cmd/go.

Alternatively we can maybe have gopls check for overlay files in the modcache before calling packages.Load and filtering those out.

findleyr · 2025-02-18T16:19:40Z

Gopls can definitely filter out overlays from the module cache (perhaps reporting an error back to the user). Then it would be safe for the Go command to fail in that case.

gopherbot · 2025-02-18T21:48:08Z

Change https://go.dev/cl/650475 mentions this issue: cmd/go: explicitly reject overlays affecting GOMODCACHE

The go command assumes that GOMODCACHE is immutable. As an example of one place the assumption is made, the modindex won't stat the files in GOMODCACHE when getting the cache key for the index entry and just uses the path of the module in the modcache (basically the module's name and version). Explicitly reject overlays affecting GOMODCACHE to avoid surprising and incorrect behavior. For #71783 For #71075 Change-Id: I21dd5d39d71037de473b09ac8482a1867864e11f Reviewed-on: https://go-review.googlesource.com/c/go/+/650475 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Alan Donovan <adonovan@google.com> Reviewed-by: Robert Findley <rfindley@google.com>

gopherbot added the Tools label Dec 31, 2024

gopherbot added this to the Unreleased milestone Dec 31, 2024

mateusz834 added NeedsInvestigation and removed Tools labels Dec 31, 2024

mateusz834 added the Tools label Dec 31, 2024

gabyhelp mentioned this issue Jan 2, 2025

x/tools/go/packages: overlay does not cause ExportFile to be cleared #71098

Open

hajimehoshi mentioned this issue Feb 16, 2025

cmd/go: adding a new file by -overlay didn't work in Go 1.24, though this worked in Go 1.23 #71783

Closed

findleyr mentioned this issue Feb 18, 2025

x/tools/gopls: filter out overlays in the module cache #71816

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/go/packages: overlay does not work properly with external module files #71075

x/tools/go/packages: overlay does not work properly with external module files #71075

mateusz834 commented Dec 31, 2024

gabyhelp commented Dec 31, 2024

mateusz834 commented Jan 1, 2025

mateusz834 commented Jan 1, 2025 •

edited

Loading

matloob commented Jan 10, 2025

mateusz834 commented Jan 11, 2025 •

edited

Loading

matloob commented Jan 13, 2025

mateusz834 commented Jan 13, 2025

matloob commented Jan 13, 2025

findleyr commented Feb 18, 2025 •

edited

Loading

gopherbot commented Feb 18, 2025

x/tools/go/packages: overlay does not work properly with external module files #71075

x/tools/go/packages: overlay does not work properly with external module files #71075

Comments

mateusz834 commented Dec 31, 2024

gabyhelp commented Dec 31, 2024

mateusz834 commented Jan 1, 2025

mateusz834 commented Jan 1, 2025 • edited Loading

matloob commented Jan 10, 2025

mateusz834 commented Jan 11, 2025 • edited Loading

matloob commented Jan 13, 2025

mateusz834 commented Jan 13, 2025

matloob commented Jan 13, 2025

findleyr commented Feb 18, 2025 • edited Loading

gopherbot commented Feb 18, 2025

mateusz834 commented Jan 1, 2025 •

edited

Loading

mateusz834 commented Jan 11, 2025 •

edited

Loading

findleyr commented Feb 18, 2025 •

edited

Loading