Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: internal error in go work sync #65363

Open
trajan0x opened this issue Jan 30, 2024 · 8 comments
Open

cmd/go: internal error in go work sync #65363

trajan0x opened this issue Jan 30, 2024 · 8 comments
Assignees
@matloob
Labels
FixPending Issues that have a fix which has not yet been reviewed or submitted. GoCommand cmd/go modules NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.23

Comments

@trajan0x
Copy link

trajan0x commented Jan 30, 2024 •
edited

Go version

go version go1.21.5 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/trajan0x/Library/Caches/go-build'
GOENV='/Users/trajan0x/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/trajan0x/go/pkg/mod'
GONOPROXY='github.com/synapsecns/*'
GONOSUMDB='github.com/synapsecns/*'
GOOS='darwin'
GOPATH='/Users/trajan0x/go'
GOPRIVATE='github.com/synapsecns/*'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.21.5/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.21.5/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.5'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/12/8xtw48x951g0vv4z_ctcr2hr0000gn/T/go-build1424252971=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

  1. git clone https://github.com/synapsecns/sanguine
  2. go work sync
  3. git checkout 995354fca4e89c0931ea5f6f0a77f170aac2f13e (this is happening across a couple commits and I found it on Add native ETH support to relayer synapsecns/sanguine#1840, just providing a single commit on master for reproducibility)

(Note: recipe won't work for this kinda bug, but I created a repl of the repo here and ran the command just for ease of debugging.

What did you see happen?

go: cloud.google.com/go@v0.110.7 requires
        github.com/google/s2a-go@v0.1.4
panic: internal error: found a version conflict, but no constraint it violates

goroutine 1 [running]:
cmd/go/internal/modload.editRequirements({0x1010edbd0, 0x1014830a0}, 0x14013bbaaa0, {0x0, 0x0, 0x1400bb7e828?}, {0x14007bac000, 0x108, 0x1400bb7e800?})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/internal/modload/edit.go:336 +0x2bc0
cmd/go/internal/modload.EditBuildList({0x1010edbd0, 0x1014830a0}, {0x0, 0x0, 0x0}, {0x14007bac000, 0x108, 0x200})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/internal/modload/buildlist.go:640 +0x64
cmd/go/internal/workcmd.runSync({0x1010edbd0, 0x1014830a0}, 0x140000265b8?, {0x14?, 0x1010248c0?, 0x100adbc6c?})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/internal/workcmd/sync.go:117 +0x4d0
main.invoke(0x10143fd80, {0x140000201a0, 0x1, 0x1})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/main.go:268 +0x4f0
main.main()
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/main.go:186 +0x754

What did you expect to see?

Success, another error, but not a panic.

The fact that this isn't handled gracefully tells me it's probably unintended behavior. But I'm a little unsure based on the context in edit.go
@bcmills bcmills changed the title go work sync internal error cmd/go: internal error in go work sync Jan 30, 2024
@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. GoCommand cmd/go modules labels Jan 30, 2024
@bcmills bcmills self-assigned this Jan 30, 2024
@bcmills bcmills added this to the Go1.23 milestone Jan 30, 2024
This was referenced Feb 5, 2024
Closed
Merged
@bcmills
Copy link
Contributor

bcmills commented Mar 14, 2024

Some notes:

  1. This does appear to be perfectly reproducible in synapsecns/sanguine@995354f.
  2. github.com/google/s2a-go@v0.1.4 is at go 1.16 and therefore unpruned. It is required directly by several of the modules in the workspace.

When a pruned module has a requirement on another root module, and that root is unpruned, we need to transition from the pruned side of the module graph to the unpruned side. I suspect that isn't happening when it needs to in dqTracker.path.

@bcmills bcmills added NeedsFix The path to resolution is known, but the work has not been done. FixPending Issues that have a fix which has not yet been reviewed or submitted. labels Mar 14, 2024
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 14, 2024
@gopherbot
Copy link

Change https://go.dev/cl/571800 mentions this issue: cmd/go/internal/modload: follow dependencies of unpruned roots in dqTracker.path

@bcmills bcmills removed their assignment Mar 15, 2024
@bcmills
Copy link
Contributor

bcmills commented Mar 15, 2024

The bug is triggering in the contrib/terraform-provider-helmproxy module, and the conflict is coming about because the workspace-selected version of github.com/google/pprof is ending up lower than the module-selected version — but github.com/google/pprof isn't relevant to this module anyway.

sanguine/contrib/terraform-provider-helmproxy$ go list -m github.com/google/pprof
github.com/google/pprof v0.0.0-20210506205249-923b5ab0fc1a

sanguine/contrib/terraform-provider-helmproxy$ GOWORK=off go list -m github.com/google/pprof
github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1

@bcmills
Copy link
Contributor

bcmills commented Mar 15, 2024

The dependency on the mismatched pprof comes through an old k8s.io/client-go@v0.25.5:

k8s.io/client-go@v0.25.5 cloud.google.com/go@v0.97.0
cloud.google.com/go@v0.97.0 github.com/googleapis/gax-go/v2@v2.1.0
github.com/googleapis/gax-go/v2@v2.1.0 google.golang.org/api@v0.54.0
google.golang.org/api@v0.54.0 cloud.google.com/go@v0.90.0
cloud.google.com/go@v0.90.0 github.com/google/pprof@v0.0.0-20210720184732-4bb14d4b1be1

But k8s.io/api v0.25.5 is at go 1.19, so it normally has module graph pruning enabled — it must be getting pulled in through some other dependency.

@bcmills
Copy link
Contributor

bcmills commented Mar 15, 2024

Ok, here's the actual bad path: github.com/hashicorp/terraform-provider-google/v4 v4.2.0 is explicitly required and unpruned (at go 1.16).

From there, we reach the bad pprof unpruned:

github.com/hashicorp/terraform-provider-google/v4@v4.2.0 google.golang.org/api@v0.60.0
google.golang.org/api@v0.60.0 cloud.google.com/go@v0.97.0
cloud.google.com/go@v0.97.0 github.com/googleapis/gax-go/v2@v2.1.0
github.com/googleapis/gax-go/v2@v2.1.0 google.golang.org/api@v0.54.0
google.golang.org/api@v0.54.0 cloud.google.com/go@v0.90.0
cloud.google.com/go@v0.90.0 github.com/google/pprof@v0.0.0-20210720184732-4bb14d4b1be1

So it appears that in workspace mode we are failing to load the transitive dependencies of github.com/hashicorp/terraform-provider-google/v4@v4.2.0.

@bcmills
Copy link
Contributor

bcmills commented Mar 15, 2024

contrib/tfcore/go.mod is using a wildcard replace directive to replace google.golang.org/api with the source code of a specific version of that module:

	google.golang.org/api => google.golang.org/api v0.86.0

@bcmills
Copy link
Contributor

bcmills commented Mar 15, 2024

So the skew in the versions seems to come from applying the replace directives inconsistently in go work sync.

Moreover, the error returned from EditBuildList causes go work sync to silently continue instead of erroring out:
https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/workcmd/sync.go;l=117-120;drc=ce8146ed3361f584ba79427ac6c6d6fe9c297bea

@bcmills
Copy link
Contributor

bcmills commented Mar 15, 2024 •
edited

So, to summarize:

  • There is one bug in how editRequirements applies module graph pruning when diagnosing unresolvable conflicts, which should be fixed by https://go.dev/cl/571800.
  • There is another bug in how go work sync applies replace directives, which will need a separate fix.

The combination of the two bugs is what produces the observed symptoms.

gopherbot pushed a commit that referenced this issue Mar 15, 2024
@gopherbot
cmd/go/internal/modload: follow dependencies of unpruned roots in dqT…
159144f 
…racker.path

For #65363.

Change-Id: I82ae1098b00c8772ef8d3aa92197e7d8c66d1b37
Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest,gotip-windows-amd64-longtest
Reviewed-on: https://go-review.googlesource.com/c/go/+/571800
Reviewed-by: Michael Matloob <matloob@golang.org>
Auto-Submit: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
@matloob matloob self-assigned this Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@matloob matloob

Labels
FixPending Issues that have a fix which has not yet been reviewed or submitted. GoCommand cmd/go modules NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.23
Development

No branches or pull requests
4 participants
@bcmills @gopherbot @matloob @trajan0x