Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent go1.13 pkg/mod/cache/download folder. #34866

Closed
c00w opened this issue Oct 12, 2019 · 5 comments
Closed

Inconsistent go1.13 pkg/mod/cache/download folder. #34866

c00w opened this issue Oct 12, 2019 · 5 comments

Comments

@c00w
Copy link

c00w commented Oct 12, 2019

What version of Go are you using (go version)?

go version
go version go1.13.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/tmp/b/cache"
GOENV="/tmp/b/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/tmp/b"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/nix/store/2kzdq7gyxz9jqfqcrp97z2y03vbw4glc-go-1.13.1/share/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/nix/store/2kzdq7gyxz9jqfqcrp97z2y03vbw4glc-go-1.13.1/share/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="/nix/store/7hzmz83nv8khpwsk858yaac7y3idh0a7-gcc-wrapper-8.3.0/bin/cc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build511834670=/tmp/go-build -gno-record-gcc-switches"

What did you do?

rm -r /tmp/a /tmp/b
mkdir -p /tmp/a/src
git clone https://github.com/wata727/tflint.git /tmp/a/src
cd /tmp/a/src && GOPATH=/tmp/a GOCACHE=/tmp/b/cache GOENV=/tmp/a/env go mod download
mkdir -p /tmp/b/src
git clone https://github.com/wata727/tflint.git /tmp/b/src
cd /tmp/b/src && GOPATH=/tmp/b GOCACHE=/tmp/b/cache GOENV=/tmp/b/env go mod download
diff -r /tmp/a/pkg /tmp/b/pkg

What did you expect to see?

No difference in diff

What did you see instead?

With go1.12, the diff shows no difference, with go1.13 it shows

Only in /tmp/b/pkg/mod/cache/download/sumdb/sum.golang.org/tile/8/0/x001/216.p: 31
Only in /tmp/a/pkg/mod/cache/download/sumdb/sum.golang.org/tile/8/0/x001/216.p: 52

This behavior is used by the nix build system to download and store dependencies for go programs. We are seeing inconsistent hashes with go 1.13 due to differences in the module cache related to the go sum server, which is breaking the build system for certain packages.

This appears to effect the following packages and versions.
github.com/wata727/tflint@v0.12.1
github.com/instrumenta/conftest@v0.14.0
github.com/syncthing/syncthing@1.3.0

We have some discussion in NixOS/nixpkgs#71049 and NixOS/nixpkgs#68135
DIsabling the sumdb fixes this. It looks like we're getting a new proof(?) every time for some packages(?).

@heschi
Copy link
Contributor

heschi commented Oct 12, 2019

These are partial tiles as described in https://research.swtch.com/tlog#serving_tiles. They will appear for any part of the tree that isn't completely filled out yet. That's up to half of the tree at any given time, if I remember the design correctly. The packages that are causing problems are probably just in a newer part of the tree than the rest.

I am extremely surprised that disabling the sumdb doesn't change anything. Are you absolutely sure it still downloads those files with GOSUMDB=off? Though I wouldn't necessarily recommend that -- presumably you want to verify that you've got the right files?

@c00w
Copy link
Author

c00w commented Oct 12, 2019

Sorry screwed up my negative, GOSUMDB=off fixes this, but in general we would like to use the sum proxy since it provides some nice content security.

@c00w
Copy link
Author

c00w commented Oct 13, 2019

I guess since the proof tree is always changing, we should just zero out the pkg/mod/cache/download/sumdb folder when generating the build inputs.

@heschi
Copy link
Contributor

heschi commented Oct 13, 2019

That's what I'd do. Closing, since this is all working as intended.

@heschi heschi closed this as completed Oct 13, 2019
@JSkrat
Copy link

JSkrat commented Aug 24, 2020

that still happens when install process breaks for some reason (sudden reboot for example) and that manual fix works. But shouldn't go installer detect that and fix that all by itself? Or at least with a prompt

@golang golang locked and limited conversation to collaborators Aug 24, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants