Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go mod vendor does not produce a consistent modules.txt #28102

Closed
jsternberg opened this issue Oct 9, 2018 · 10 comments
Closed

cmd/go: go mod vendor does not produce a consistent modules.txt #28102

jsternberg opened this issue Oct 9, 2018 · 10 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@jsternberg
Copy link

What did you do?

Ran go mod vendor and committed the results to source control. Then ran it in continuous integration by copying the folder, running go mod vendor, and diffing the folders with diff -qr vendor.orig/ vendor/ to find differences.

What did you expect to see?

I expected to see no differences.

What did you see instead?

There were many differences. Running diff -qr <(sort vendor.orig/modules.txt) <(sort vendor/modules.txt) caused it to report no differences between the files, so it appears the ordering was different.

System details

go version go1.11.1 darwin/amd64
GOARCH="amd64"
GOBIN="/Users/jsternberg/go/pkg/bin/github.com/influxdata/idpe"
GOCACHE="/Users/jsternberg/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jsternberg/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.1/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jsternberg/go/src/github.com/influxdata/idpe/go.mod"
GOROOT/bin/go version: go version go1.11.1 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.11.1
uname -v: Darwin Kernel Version 18.0.0: Wed Aug 22 20:13:40 PDT 2018; root:xnu-4903.201.2~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.14
BuildVersion:	18A391
lldb --version: lldb-1000.11.37.1
  Swift-4.2
@bcmills
Copy link
Contributor

bcmills commented Oct 9, 2018

Running diff -qr <(sort vendor.orig/modules.txt) <(sort vendor/modules.txt) caused it to report no differences between the files, so it appears the ordering was different.

Different in what way? (Which modules were transposed, and does one of them transitively depend on the other?)

@gopherbot
Copy link

Change https://golang.org/cl/140862 mentions this issue: cmd/go/internal/mvs: document that BuildList is sorted

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. modules labels Oct 10, 2018
@bcmills bcmills added this to the Go1.12 milestone Oct 10, 2018
@bcmills
Copy link
Contributor

bcmills commented Oct 10, 2018

It appears that the BuildList itself is sorted. If there is any nondeterminism, it likely comes from concurrent import scanning, but I haven't been able to reproduce it in a unit-test so far.

@jsternberg
Copy link
Author

I'll work on a reproducer for this today. If it helps you, it seems to occur for me when I use go mod vendor on a Mac, then I run it again within the golang:1.11 image in Docker (so on Linux). The version of 1.11.0 or 1.11.1 don't seem to make a difference.

@bcmills
Copy link
Contributor

bcmills commented Oct 10, 2018

Ah. If you repeatedly run go mod vendor on a single platform, do you get consistent results?

@jsternberg
Copy link
Author

Yes, I think I do.

I'll attempt to verify with an example on a public repo when I can today or at least get more information on the difference in the private repo.

@jsternberg
Copy link
Author

I tried to produce a test to reproduce this and I found a combination of user-failure/a potentially additional command as needed.

The reason why I got the failure is because as part of CI, we were doing this:

$ cp -a vendor/ vendor.orig/
$ go mod vendor

When I copy vendor to another directory, it becomes part of the current project in some way and it starts resolving all of the vendored dependencies as if they were part of the current project. So I think my initial issue is wrong as that's user failure.

But, the reason why we do that is because in our CI, the place where we run that doesn't have git. We commit our vendor directory and we want to ensure our vendor directory is correctly committed. Our CI doesn't have git at this point, so I copied the vendor directory to contain it. It doesn't appear like there's a way to validate the vendor directory similar to dep check.

Am I missing a command or should I create a separate issue for that?

@mark-rushakoff
Copy link
Contributor

From go help packages:

Directory and file names that begin with "." or "_" are ignored
by the go tool, as are directories named "testdata".

_vendor.orig or .vendor.orig should keep go from thinking the copies are part of the project.

@bcmills
Copy link
Contributor

bcmills commented Oct 11, 2018

It doesn't appear like there's a way to validate the vendor directory similar to dep check.

I think that's #27348.

@bcmills
Copy link
Contributor

bcmills commented Oct 11, 2018

Closing, on the theory that @mark-rushakoff's workaround should address your immediate needs and #27348 should provide a longer-term solution.

Please let me know if there are any other parts that we need to keep track of.

@bcmills bcmills closed this as completed Oct 11, 2018
gopherbot pushed a commit that referenced this issue Oct 26, 2018
Updates #28102

Change-Id: Iee1ff64c7720108d6d26bfbff60ea51877093960
Reviewed-on: https://go-review.googlesource.com/c/140862
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
@golang golang locked and limited conversation to collaborators Oct 11, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

4 participants