cmd/go: go list fails with submodules which have test-only dependencies #60667

smira · 2023-06-07T19:16:15Z

What version of Go are you using (`go version`)?

$ go version
go1.20.5

Does this issue reproduce with the latest release?

yes, I believe it is related to #60001

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/smira/.cache/go-build"
GOENV="/home/smira/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/smira/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/smira/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/smira/sdk/go1.20.5"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/smira/sdk/go1.20.5/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.5"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/smira/Documents/go-test-package-failure/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build337276505=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I hit this on a big repository, but I reduced this down to https://github.com/smira/go-test-package-failure.

There's a submodule which has an import of a package from _test.go file.

What did you expect to see?

Switching to older version of Go "fixes" it:

$ ~/go/bin/go1.20.3 list -u -m -json all
{
	"Path": "github.com/smira/go-test-package-failure",
	"Main": true,
	"Dir": "/home/smira/Documents/go-test-package-failure",
	"GoMod": "/home/smira/Documents/go-test-package-failure/go.mod",
	"GoVersion": "1.20"
}
{
	"Path": "github.com/santhosh-tekuri/jsonschema/v5",
	"Version": "v5.3.0",
	"Time": "2023-04-04T18:54:33Z",
	"Indirect": true,
	"Dir": "/home/smira/go/pkg/mod/github.com/santhosh-tekuri/jsonschema/v5@v5.3.0"
}
{
	"Path": "github.com/smira/go-test-failure/subpackage",
	"Version": "v0.0.0-00010101000000-000000000000",
	"Replace": {
		"Path": "./subpackage",
		"Dir": "/home/smira/Documents/go-test-package-failure/subpackage",
		"GoMod": "/home/smira/Documents/go-test-package-failure/subpackage/go.mod",
		"GoVersion": "1.20"
	},
	"Dir": "/home/smira/Documents/go-test-package-failure/subpackage",
	"GoMod": "/home/smira/Documents/go-test-package-failure/subpackage/go.mod",
	"GoVersion": "1.20"
}

What did you see instead?

In the checkout, the following fails:

$ ~/go/bin/go1.20.5 list -u -m -json all
go: updates to go.sum needed, disabled by -mod=readonly

The text was updated successfully, but these errors were encountered:

smira · 2023-06-07T19:23:58Z

Looking more into it, it might be related to github.com/santhosh-tekuri/jsonschema/v5 package, but I'm not sure what's exactly wrong with it

smira · 2023-06-08T10:41:01Z

Found a hot fix

I believe go mod tidy would remove this from go.mod, as it's unused.

smira · 2023-06-08T13:03:43Z

@Nasfame I can't reproduce it, go mod tidy removes the line from go.sum and things keep failing.

Basically the bug is that go mod tidy and go list are in disagreement on what should be in go.sum.

dr2chase · 2023-06-08T14:04:53Z

@bcmills @ianlancetaylor @rsc

bcmills · 2023-06-08T15:47:45Z

Duplicate of #56222

smira · 2023-06-08T15:55:11Z

I'm not sure why it's a duplicate, the problem comes only with #56222 fix backported to Go 1.20 in Go 1.20.5. It worked fine before that.

bcmills · 2023-06-08T16:06:22Z

Ah, I had missed that detail. Yes, it seems that the backported fix needs an extra condition for when to save the checksums.

lespea · 2023-06-08T19:08:02Z

I see this is marked for 1.21, when this is resolved will this be backported to 1.20.6? This is breaking a lot of our tooling and we won't be able to update to 1.21+ for the foreseeable future giving that go is dropping support for a lot of operating system versions.

bcmills · 2023-06-08T19:15:17Z

Yes, I intend for it to be. (That, or we will roll back part of all of #60001 and #60000.)

gopherbot · 2023-06-08T22:12:37Z

Change https://go.dev/cl/502015 mentions this issue: cmd/go: omit checksums for go.mod files needed for go version lines more often in pre-1.21 modules

bcmills · 2023-06-08T22:12:49Z

@gopherbot, please backport to Go 1.19 and 1.20. This is a regression introduced in a previous patch release.

gopherbot · 2023-06-08T22:13:07Z

Backport issue(s) opened: #60697 (for 1.19), #60698 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

bcmills · 2023-06-08T22:18:16Z

In the meantime you can work around this with one of the following:

build from source and patch in the proposed fix CLs (once I've resolved the merge conflicts for the backports)
set -mod=mod in the go list command (to allow it to update the go.sum file)
run go mod tidy -compat=1.16 (instead of just go mod tidy) to record all of the needed checksums upfront.

gopherbot · 2023-06-08T22:19:13Z

Change https://go.dev/cl/502016 mentions this issue: [release-branch.go1.20] cmd/go: omit checksums for go.mod files needed for go version lines more often in pre-1.21 modules

gopherbot · 2023-06-09T01:28:59Z

Change https://go.dev/cl/502017 mentions this issue: [release-branch.go1.19] cmd/go: omit checksums for go.mod files needed for go version lines more often in pre-1.21 modules

…d for go version lines more often in pre-1.21 modules This updates the logic from CL 489075 to avoid trying to save extra sums if they aren't already expected to be present and cfg.BuildMod != "mod" (as in the case of "go list -m -u all" with a go.mod file that specifies go < 1.21). Fixes #60698. Updates #60667. Updates #56222. Change-Id: Ied6ed3e80a62f9cd9a328b43a415a42d14481056 Reviewed-on: https://go-review.googlesource.com/c/go/+/502016 Reviewed-by: Russ Cox <rsc@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com>

…d for go version lines more often in pre-1.21 modules This updates the logic from CL 489075 to avoid trying to save extra sums if they aren't already expected to be present and cfg.BuildMod != "mod" (as in the case of "go list -m -u all" with a go.mod file that specifies go < 1.21). Fixes #60697. Updates #60667. Updates #56222. Change-Id: Ied6ed3e80a62f9cd9a328b43a415a42d14481056 Reviewed-on: https://go-review.googlesource.com/c/go/+/502017 Reviewed-by: Michael Matloob <matloob@golang.org> Reviewed-by: Russ Cox <rsc@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Bypass: Bryan Mills <bcmills@google.com>

dmitshur changed the title ~~go: go list fails with submodules which have test-only dependencies~~ cmd/go: go list fails with submodules which have test-only dependencies Jun 7, 2023

dr2chase added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 8, 2023

dmitshur added the GoCommand cmd/go label Jun 8, 2023

bcmills marked this as a duplicate of #56222 Jun 8, 2023

bcmills closed this as not planned Won't fix, can't repro, duplicate, stale Jun 8, 2023

bcmills reopened this Jun 8, 2023

bcmills self-assigned this Jun 8, 2023

bcmills added modules NeedsFix The path to resolution is known, but the work has not been done. labels Jun 8, 2023

bcmills added this to the Go1.21 milestone Jun 8, 2023

gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 8, 2023

This was referenced Jun 8, 2023

cmd/go: go list fails with submodules which have test-only dependencies [1.19 backport] #60697

Closed

cmd/go: go list fails with submodules which have test-only dependencies [1.20 backport] #60698

Closed

bcmills added the release-blocker label Jun 9, 2023

joergjo added a commit to joergjo/go-samples that referenced this issue Jun 9, 2023

fix: go.sum to avoid golang/go#60667

7698483

gopherbot closed this as completed in 1db2377 Jun 12, 2023

bcmills mentioned this issue Jun 12, 2023

cmd/go/internal/modload: clean up checksum-saving logic #60748

Open

lespea mentioned this issue Jun 18, 2023

Error: disabled by -mod=readonly oligot/go-mod-upgrade#39

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: go list fails with submodules which have test-only dependencies #60667

cmd/go: go list fails with submodules which have test-only dependencies #60667

smira commented Jun 7, 2023

smira commented Jun 7, 2023

smira commented Jun 8, 2023

smira commented Jun 8, 2023

dr2chase commented Jun 8, 2023

bcmills commented Jun 8, 2023

smira commented Jun 8, 2023

bcmills commented Jun 8, 2023

lespea commented Jun 8, 2023

bcmills commented Jun 8, 2023

gopherbot commented Jun 8, 2023

bcmills commented Jun 8, 2023

gopherbot commented Jun 8, 2023

bcmills commented Jun 8, 2023

gopherbot commented Jun 8, 2023

gopherbot commented Jun 9, 2023

cmd/go: go list fails with submodules which have test-only dependencies #60667

cmd/go: go list fails with submodules which have test-only dependencies #60667

Comments

smira commented Jun 7, 2023

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

smira commented Jun 7, 2023

smira commented Jun 8, 2023

smira commented Jun 8, 2023

dr2chase commented Jun 8, 2023

bcmills commented Jun 8, 2023

smira commented Jun 8, 2023

bcmills commented Jun 8, 2023

lespea commented Jun 8, 2023

bcmills commented Jun 8, 2023

gopherbot commented Jun 8, 2023

bcmills commented Jun 8, 2023

gopherbot commented Jun 8, 2023

bcmills commented Jun 8, 2023

gopherbot commented Jun 8, 2023

gopherbot commented Jun 9, 2023

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?