Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: removing/switching tags on repository causes a go module checksum mismatch error #33969

Closed
rekby opened this issue Aug 30, 2019 · 7 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@rekby
Copy link

rekby commented Aug 30, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.9 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/rekby/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/rekby/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.12.9/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12.9/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/rekby/tmp/gomod-test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/8_/68c7r6rj6xg4fpjls__qtsjn18s1hz/T/go-build805875439=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

  1. Create repo github.com/rekby-forks/test-mod
  2. Create some file and tag it as semver, for example v0.0.2
  3. Create new go module
    go.mod
module rekby
go 1.12
require github.com/rekby-forks/test-mod v0.0.2

go.sum

package main

import tst "github.com/rekby-forks/test-mod"
import "fmt"

func main(){
    fmt.Println(tst.Test())
}
  1. go build. On it step created file go.sum
github.com/rekby-forks/test-mod v0.0.2 h1:ks22upaESPuYtz3t4FmkCKQCse5oMPs/gVTRavFVJlY=
github.com/rekby-forks/test-mod v0.0.2/go.mod h1:awjpQNGe4nVJmjCzjODnLPXGpp3acr9evJveSVjfBOY=
  1. go to repo github.com/rekby-forks/test-mod and remove tag, or remove and put for other commit.

  2. clean go cache

rm -rf /Users/rekby/Library/Caches/go-build/*; rm -rf ~/go/pkg/
  1. go build

What did you expect to see?

Build same binary, which as on step 4 (need commit exist in github.com/rekby-forks/test-mod)

What did you see instead?

if remove tag:

go: finding github.com/rekby-forks/test-mod v0.0.2
go: github.com/rekby-forks/test-mod@v0.0.2: unknown revision v0.0.2
go: error loading module requirements

if switch tag to other commit

go: finding github.com/rekby-forks/test-mod v0.0.2
go: downloading github.com/rekby-forks/test-mod v0.0.2
verifying github.com/rekby-forks/test-mod@v0.0.2: checksum mismatch
	downloaded: h1:PavIxURHqy6UUFJlEjFkimV/fJYUwPThVMa0MPvnBX0=
	go.sum:     h1:ks22upaESPuYtz3t4FmkCKQCse5oMPs/gVTRavFVJlY=

The commit exist in repo, it was tag change only. But I can't build binary and can't know what commit I need. Only way - manually (or self script) check every commit.

I think go.sum must contain info about revision of VSC (revision on svn, hash commit for git, hg, etc) and be independent from tags (which can beeasy change/remove).

@odeke-em odeke-em changed the title Bad external dependency description cmd/go: removing/switching tags on repository causes a go module checksum mismatch error Aug 31, 2019
@odeke-em
Copy link
Member

odeke-em commented Sep 1, 2019

Thank you for reporting this issue @rekby!

I am not sure what we can do here since the previous tag no longer exists upstream on the repository, meanwhile your go.mod/go.sum files contain the old checksums, thus you've declared that you want the non-existent commit to be built. This can happen in cases such as if someone
nefariously rewrites a commit and pushes up, so that error that you got is a safeguard.
Perhaps you should run go mod tidy on it to get back the old commit along with a go get -u?

However, I shall kindly loop in some go mod experts @bcmills @jayconrod.

@odeke-em odeke-em added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 1, 2019
@rekby
Copy link
Author

rekby commented Sep 2, 2019

When commit have no tag - go.sum contains part of commit hash - and it allow to computer find commit independent of tags.
Tags - more readable label for human and if go.sum contain tag - it is more readable, but lost accuracy - because the tag easy can rewrite by repo author and same tag point to other commit.

With commit it is more difficult - rewrite repo history and force push will change commit hashes in git and need full rebuild repo for svn.

I suggest add revision/commit hash to in additional to tag names, for example:

github.com/rekby-forks/test-mod v0.0.2(e9a220f8f3579aa9613bf755381b88f1fff60716) h1:ks22upaESPuYtz3t4FmkCKQCse5oMPs/gVTRavFVJlY=

it allow human read tag and human/computer read hash.

Then if tag point to different commit - go build can error with question what to do (change hash, change tag, stop) or fail with description with variants: change hash for use actual tagged version, change tag - for use previous commit.
Or it can show warning and use hash for checkout.

In any varian commit hash allow manual undestand what was previous state.
And prevent situation, when commit exist in repo, but no way to find it.

@bcmills
Copy link
Contributor

bcmills commented Sep 3, 2019

The go.mod and go.sum files are both written in terms of semantic versions, not commit hashes. That makes them — and the module proxy protocol — agnostic to the particular version-control system used to manage the code. (The commit hashes still leak through as pseudo-versions, but even then they're still represented as semantically-meaningful semver strings.)

The overall design of modules is that versions are immutable. A given version of a given module must always refer to the exact same contents. (The checksum database, for which support was added in Go 1.13, helps to ensure that property.) For that reason, we explicitly do not support changing semantic-version tags to point to different contents over time — if you need to tag a new release, it should have a new version number.

If you are worried about particular versions of your module dependencies disappearing or being deleted, one mitigation is to use a durable module mirror instead of fetching directly from upstream version control.

@bcmills
Copy link
Contributor

bcmills commented Sep 3, 2019

@rekby, did you encounter this problem in the wild, or is this concern mostly theoretical?

We've tried to communicate clearly that module versions must be immutable, but perhaps there are some loose ends in the documentation that we can clear up if we can figure out the source of confusion.

@bcmills bcmills added modules WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Sep 3, 2019
@rekby
Copy link
Author

rekby commented Sep 4, 2019

@bcmills it is mote theoretical problem - about repo owner.

For example I can change some tags in my small popular (near private) repo some times for ьн сщтмутшутсу if I think about only I use it. Or if I typo in tag (for example set tag v0.2 instead of v0.0.2). If I mistake and somebody add my repo from error tag v0.2 in short time, what tag existed - it can be problem for code consumer.

I did not encounded the problem in real wild.

@bcmills
Copy link
Contributor

bcmills commented Sep 4, 2019

You may safely change the tags on your repository as long as you are certain that nobody — including sum.golang.org — has fetched those tags yet.

If you accidentally typo a tag, such as v0.2.1 instead of v0.1.2, then the most robust way to fix it is to select and publish a higher version still.

@bcmills
Copy link
Contributor

bcmills commented Sep 4, 2019

I don't think there's anything more to be done about this. The entire purpose of the go.sum file and checksum database is to enforce that the mapping of versions to source code does not change over time.

@bcmills bcmills closed this as completed Sep 4, 2019
@golang golang locked and limited conversation to collaborators Sep 3, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

4 participants