Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go.mod inconsistency between "go mod tidy" and "go build" #39570

Closed
ceejatec opened this issue Jun 13, 2020 · 11 comments
Closed

cmd/go: go.mod inconsistency between "go mod tidy" and "go build" #39570

ceejatec opened this issue Jun 13, 2020 · 11 comments
Labels
FrozenDueToAge GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@ceejatec
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.14.4 linux/amd64

Does this issue reproduce with the latest release?

Yes (confirmed with gotip).

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ceej/.cache/go-build"
GOENV="/home/ceej/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ceej/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ceej/.cbdepscache/exploded/x86_64/go-1.14.4/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/ceej/.cbdepscache/exploded/x86_64/go-1.14.4/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/foo/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build286401846=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I have a project with two inter-dependent modules. Our overall build checks out both modules into sibling directories, and we use "replace" directives so that the code can find each other. This is necessary because both projects are actively developed and we don't want to have to release/tag each of them every time there's a change to get into a build.

One of those projects also depends on a separate package "go-jsonpointer" which is not yet converted to Go modules, and that package in turn depends on another package "gojson" which is also not yet converted to Go modules.

What did you expect to see?

We expect that "go mod tidy" immediately followed by "go build -mod=read" should succeed. This was confirmed by @jayconrod to be expected behaviour on #31248 .

What did you see instead?

What we see is that "go build" updates go.mod to add "gojson" as an //indirect dependency - this is correct. However, "go mod tidy" removes that entry from go.mod. What this means is that if we run "go mod tidy" (which is normal good practice), then "go build -mod=readonly" will fail with

/home/ceej/go/pkg/mod/github.com/dustin/go-jsonpointer@v0.0.0-20160814072949-ba0abeacc3dc/bytes.go:9:2: cannot find module providing package github.com/dustin/gojson: import lookup disabled by -mod=readonly

Minimal reproducer

I've created a minimal test case which demonstrates this issue, including a script that shows the error when using "go build -mod=readonly": https://github.com/ceejatec/go-mod-bug

You can see the bug by just:

git clone git://github.com/ceejatec/go-mod-bug
cd go-mod-bug
./repro.sh
@ceejatec
Copy link
Author

On #29773, @bcmills said that "go mod tidy" will include

  • all of the direct dependencies, plus
  • the subset of indirect dependencies that have been upgraded past their otherwise-implied versions.

I suspect that this logic is getting thrown off in this case by the fact that "foo" refers to itself using a "replace" directive. However, this replace is necessary given the way our modules are developed. foo depends on bar, while bar depends on foo. bar mentions foo with a replace directive in its own go.mod, but when building foo, only foo's go.mod is referenced. Therefore, for bar's reference to foo to resolve, foo/go.mod has to include a replace directive for itself.

@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 13, 2020
@dmitshur dmitshur added this to the Backlog milestone Jun 13, 2020
@dmitshur dmitshur changed the title go.mod inconsistency between "go mod tidy" and "go build" cmd/go: go.mod inconsistency between "go mod tidy" and "go build" Jun 13, 2020
@golang golang deleted a comment Jun 14, 2020
@bcmills
Copy link
Contributor

bcmills commented Jun 15, 2020

Duplicate of #34417

@bcmills bcmills marked this as a duplicate of #34417 Jun 15, 2020
@bcmills bcmills closed this as completed Jun 15, 2020
@bcmills
Copy link
Contributor

bcmills commented Jun 15, 2020

(See specifically #34417 (comment).)

@ceejatec
Copy link
Author

Request this ticket be re-opened. It does not describe the same issue as #34417. It is true that if #34417 is acted upon as written, that this ticket will be moot. However I do not think #34417 should be implemented as written, which would make this ticket relevant.

@jayconrod
Copy link
Contributor

@ceejatec I don't understand how this issue is different from #34417. Could you elaborate? In both cases, there are two modules that require each other and are developed together. In both cases, the main module replaces itself, which is not handled correctly by go mod tidy or other commands.

I understand the solutions suggested in #34417 aren't necessarily satisfying, but unless this is a fundamentally different issue, it would be better to focus discussion there rather than splitting the conversation.

@ceejatec
Copy link
Author

The reported symptoms are similar although not the same. However the subject line for #34417 says that the goal is to remove the ability to point a "replace" directive at the module directory itself, which is quite at odds with the goal of this ticket.

@jayconrod
Copy link
Contributor

The reported symptoms are similar although not the same.

How are they different?

The original title of that issue was "go.mod changes on each build, -mod=readonly can never work", which sounds like the same symptoms. The repros people have posted seem structurally similar.

@ceejatec
Copy link
Author

They are more similar than I thought, true - the modules which are getting removed and re-added in my reproducer are separate third-party ones, but it seems that's also the case on #34417 which wasn't clear from the description.

Again, though, the big difference IMHO is the goal. That ticket is discussing the correctness of removing the feature entirely. Do you believe it is implied that if the feature is NOT removed, then the underlying bugs with "go mod tidy" will be fixed?

@bcmills
Copy link
Contributor

bcmills commented Jun 17, 2020

@ceejatec, replacing the selected version of the main module itself is not a “feature” — it is a bug.

You can replace specific older versions of the main module regardless of #34417.

@ceejatec
Copy link
Author

It may be a bug, but as these tickets show, it is also a workaround to other bugs, because the go compiler does not honor "replace" directives other than at the top level.

@jayconrod
Copy link
Contributor

Ignoring replace directives outside the main module is not a bug: the ecosystem couldn't scale if there were lots of modules with contradictory replace directives.

In any case, if you disagree with the approach in #34417, please explain further in #34417. I'd rather not split the discussion further.

@golang golang locked and limited conversation to collaborators Jun 17, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants