Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/vgo: possible non-determinism in updating go.sum #26310

Closed
daaku opened this issue Jul 10, 2018 · 6 comments
Closed

x/vgo: possible non-determinism in updating go.sum #26310

daaku opened this issue Jul 10, 2018 · 6 comments
Labels
FrozenDueToAge
Milestone
vgo

Comments

@daaku
Copy link

daaku commented Jul 10, 2018

What version of Go are you using (go version)?

go version go1.10.3 darwin/amd64 vgo:2018-02-20.1

Does this issue reproduce with the latest release?

I'm using the current latest released go and golang/vgo@cdd82a7.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/naitik/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/naitik/usr/go"
GOPROXY=""
GORACE=""
GOROOT="/Users/naitik/usr/go1.10.3"
GOTMPDIR=""
GOTOOLDIR="/Users/naitik/usr/go1.10.3/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/fn/8xcrnptx0f75fryl8n30kbyr0000gn/T/go-build924712898=/tmp/go-build -gno-record-gcc-switches -fno-common"
VGOMODROOT=""

What did you do?

I knew there was an update to one of my dependencies (crawshaw.io/sqlite) and ran vgo get -u to pull in that and any other updates.

What did you expect to see?

I expected to see changed revisions as well as updated cryptographic checksums in go.sum.

What did you see instead?

go.sum shows me new entries for the updated revisions, but the checksums are the same as the old ones. Here's the git diff after running vgo get -u:

diff --git a/go.sum b/go.sum
index e36613c..63d7df0 100644
--- a/go.sum
+++ b/go.sum
@@ -1,4 +1,5 @@
 crawshaw.io/sqlite v0.0.0-20180510162726-4e3e3e8c528e/go.mod h1:BZaitnE9BVpocOuCdi/y5XReJMUelG53e/rDSLwSFzY=
+crawshaw.io/sqlite v0.0.0-20180710105110-95479735d36f/go.mod h1:BZaitnE9BVpocOuCdi/y5XReJMUelG53e/rDSLwSFzY=
 github.com/NYTimes/gziphandler v0.0.0-20180227064818-5032c8878b9d/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
 github.com/blang/semver v0.0.0-20170727064818-2ee87856327b/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
 github.com/daaku/ensure v0.0.0-20180215095550-9bd7e49e586d/go.mod h1:Ow4+0ZvWSGA0FkpjHd3H5R0LiKuSXFBf7oBPzJpCmSk=
@@ -12,9 +13,12 @@ github.com/gorilla/csrf v1.5.1/go.mod h1:HTDW7xFOO1aHddQUmghe9/2zTvg7AYCnRCs7MxT
 github.com/gorilla/securecookie v1.1.1/go.mod h1:ra0sb63/xPlUeL+yeDciTfxMRAA+MP+HVt/4epWDjd4=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/rakyll/statik v0.1.1/go.mod h1:OEi9wJV/fMUAGx1eNjq75DKDsJVuEv1U0oYdX6GX8Zs=
+github.com/rakyll/statik v0.1.2/go.mod h1:OEi9wJV/fMUAGx1eNjq75DKDsJVuEv1U0oYdX6GX8Zs=
 github.com/valyala/bytebufferpool v0.0.0-20160817181652-e746df99fe4a/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
 github.com/valyala/quicktemplate v0.0.0-20180430205211-a91e0946457b/go.mod h1:uX8yrAzr955BRr+rAerWX4V1NHb8bFfsePffuN5lFso=
 github.com/vincent-petithory/dataurl v0.0.0-20160330182126-9a301d65acbb/go.mod h1:FHafX5vmDzyP+1CQATJn7WFKc9CvnvxyvZy6I1MrG/U=
 golang.org/x/image v0.0.0-20171214225156-12117c17ca67/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
+golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
 golang.org/x/net v0.0.0-20180524181706-dfa909b99c79/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180710023853-292b43bbf7cb/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 xojoc.pw/useragent v0.0.0-20170215185434-52903803fc66/go.mod h1:71om/Qz9HbIEjbUrkrzmJiF26FSh6tcwqSFdBBkLtJQ=

Next I ran vgo install ./... inside my module, and I saw go.sum was modified again, this time new checksums showed up for some of the entries (notice golang.org/x/net was not changed):

diff --git a/go.sum b/go.sum
index e36613c..847db9e 100644
--- a/go.sum
+++ b/go.sum
@@ -1,4 +1,6 @@
 crawshaw.io/sqlite v0.0.0-20180510162726-4e3e3e8c528e/go.mod h1:BZaitnE9BVpocOuCdi/y5XReJMUelG53e/rDSLwSFzY=
+crawshaw.io/sqlite v0.0.0-20180710105110-95479735d36f h1:vn2WsFRW6O68ppjB8cvQAMVtzUytvh+RNx9ZM+Lx4iE=
+crawshaw.io/sqlite v0.0.0-20180710105110-95479735d36f/go.mod h1:BZaitnE9BVpocOuCdi/y5XReJMUelG53e/rDSLwSFzY=
 github.com/NYTimes/gziphandler v0.0.0-20180227064818-5032c8878b9d/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
 github.com/blang/semver v0.0.0-20170727064818-2ee87856327b/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
 github.com/daaku/ensure v0.0.0-20180215095550-9bd7e49e586d/go.mod h1:Ow4+0ZvWSGA0FkpjHd3H5R0LiKuSXFBf7oBPzJpCmSk=
@@ -12,9 +14,14 @@ github.com/gorilla/csrf v1.5.1/go.mod h1:HTDW7xFOO1aHddQUmghe9/2zTvg7AYCnRCs7MxT
 github.com/gorilla/securecookie v1.1.1/go.mod h1:ra0sb63/xPlUeL+yeDciTfxMRAA+MP+HVt/4epWDjd4=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/rakyll/statik v0.1.1/go.mod h1:OEi9wJV/fMUAGx1eNjq75DKDsJVuEv1U0oYdX6GX8Zs=
+github.com/rakyll/statik v0.1.2 h1:ckeqyWXP9hT/2u8Xfi4L02B2RFlqr6fdedIPgohOAHM=
+github.com/rakyll/statik v0.1.2/go.mod h1:OEi9wJV/fMUAGx1eNjq75DKDsJVuEv1U0oYdX6GX8Zs=
 github.com/valyala/bytebufferpool v0.0.0-20160817181652-e746df99fe4a/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
 github.com/valyala/quicktemplate v0.0.0-20180430205211-a91e0946457b/go.mod h1:uX8yrAzr955BRr+rAerWX4V1NHb8bFfsePffuN5lFso=
 github.com/vincent-petithory/dataurl v0.0.0-20160330182126-9a301d65acbb/go.mod h1:FHafX5vmDzyP+1CQATJn7WFKc9CvnvxyvZy6I1MrG/U=
 golang.org/x/image v0.0.0-20171214225156-12117c17ca67/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
+golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81 h1:00VmoueYNlNz/aHIilyyQz/MHSqGoWJzpFv/HW8xpzI=
+golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
 golang.org/x/net v0.0.0-20180524181706-dfa909b99c79/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180710023853-292b43bbf7cb/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 xojoc.pw/useragent v0.0.0-20170215185434-52903803fc66/go.mod h1:71om/Qz9HbIEjbUrkrzmJiF26FSh6tcwqSFdBBkLtJQ=

Something seems to be off?
@gopherbot gopherbot added this to the vgo milestone Jul 10, 2018
@daaku
Copy link
Author

daaku commented Jul 10, 2018

To make things even more odd, I removed $GOPATH/{pkg,src/mod} and go.sum and ran vgo install ./... and found go.sum had new entries indicating that the contents of the pkg or src/mod directories were affecting go.sum in some surprising way.

@daaku
Copy link
Author

daaku commented Jul 10, 2018

Just to provide some more context, I'm seeing some unpredictable (possibly racy?) behavior between the two lines in go.sum. The first line without the /go.mod bit seems to show up when building the same piece of code without any other apparent changes. This doesn't happen with all dependencies, just some.

+github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=

@oiooj oiooj added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 10, 2018
@oiooj
Copy link
Member

oiooj commented Jul 10, 2018 •
edited

vgo updated recently，and add the /go.mod suffix on the version:

The hash is a checksum over the one-file tree containing only the go.mod for the module now. It is normal that the checksum not changed, but the version changed.

@oiooj oiooj removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 10, 2018
@rsc
Copy link
Contributor

rsc commented Jul 10, 2018

The details reported so far seems to be working correctly to me. What you're probably missing is that vgo goes out of its way to avoid downloading information until that information is necessary. This is an important optimization.

vgo get -u updates go.mod to reflect the new modules (and their dependencies) but does not run a build of everything in the module. To do the update of go.mod it only needs other go.mod files. It does not need the full source trees for the new dependencies, and so it does not look at them or even necessarily download them, and so it does not put hashes in go.sum. The /go.mod entries in the go.sum are the hashes of just the go.sum files, not the full trees.

'vgo install ./...' after 'vgo get -u' does need the source code for the new dependencies, because now it's actually reading the source code. That's why it adds new entries for some directory trees to go.sum.

If a go.mod says it requires foo.com/bar and no import found during the build points into foo.com/bar, then vgo will not load that module's source code at all. That's why not all /go.mod entries in go.sum have corresponding non-/go.mod entries, even after vgo install ./....

After removing $GOPATH/{pkg,src/mod} and go.sum and re-running vgo install ./..., I don't know exactly what you mean by "new entries". Yes, go.sum will be repopulated. It should not have any entries that were not present in the removed copy, though, since the vgo install ./... should be looking at the same inputs as before. If by "new entries" you mean something that wasn't in the go.sum you removed, I'd very much like to hear more about the details.

I don't understand the comment about go-spew. What command are you running that adds that line to go.sum? Does it happen every time you run the command? Just half the time? If the behavior is non-deterministic, can you say what setup is required to reproduce the non-determinism?

Thanks.

@rsc rsc changed the title x/vgo: go.sum cryptographic checksum incorrect right after vgo get -u x/vgo: possible non-determinism in updating go.sum Jul 10, 2018
@daaku
Copy link
Author

daaku commented Jul 10, 2018

Gotcha, I was expecting the go.sum file to be affected by vgo get -u in a way that now I understand is not what it's designed to do.

I was still experiencing some kind of non-determinism as best as I could tell. I will see if I can come up with a repro case tomorrow.

@daaku
Copy link
Author

daaku commented Jul 11, 2018

Closing this since it was due to my misunderstanding of how/when go.sum gets populated.

Specifically my build process involves a vgo run of some otherwise ignored files as well as vgo test both of which triggered pulling in dependencies which the initial vgo install did not. This is what was causing go.sum to get updated.

@daaku daaku closed this as completed Jul 11, 2018
@golang golang locked and limited conversation to collaborators Jul 11, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
vgo
Development

No branches or pull requests
4 participants
@daaku @rsc @oiooj @gopherbot