Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: missing sum after updating a different package #44129

Closed
jayconrod opened this issue Feb 5, 2021 · 44 comments · Fixed by cyberark/secretless-broker#1434
Closed

cmd/go: missing sum after updating a different package #44129

jayconrod opened this issue Feb 5, 2021 · 44 comments · Fixed by cyberark/secretless-broker#1434
Labels
FrozenDueToAge modules NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone

Comments

@jayconrod
Copy link
Contributor

jayconrod commented Feb 5, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16rc1 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jayconrod/Library/Caches/go-build"
GOENV="/Users/jayconrod/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/jayconrod/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/jayconrod/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/go/installed"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/go/installed/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16rc1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jayconrod/Code/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/rq/x0692kqj6ml8cvrhcqh5bswc008xj1/T/go-build528761276=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

# Populate go.sum
go mod tidy

# Update a transitive dependency
go get -d google.golang.org/protobuf/proto

# Build package in main module
go build

-- go.mod --
module use

go 1.15

require (
	github.com/gogo/status v1.1.0
	google.golang.org/grpc v1.12.0
)
-- use.go --
package use

import (
	_ "github.com/gogo/status"
	_ "google.golang.org/grpc/codes"
)

What did you expect to see?

The package in the main module should still build after updating one of its transitive imports to a newer, compatible version.

What did you see instead?

../../go/pkg/mod/github.com/gogo/status@v1.1.0/status.go:37:2: missing go.sum entry for module providing package github.com/golang/protobuf/ptypes/any (imported by github.com/gogo/status); to add:
	go get github.com/gogo/status@v1.1.0
../../go/pkg/mod/github.com/gogo/status@v1.1.0/status.go:38:2: missing go.sum entry for module providing package google.golang.org/genproto/googleapis/rpc/status (imported by github.com/gogo/status); to add:
	go get github.com/gogo/status@v1.1.0
../../go/pkg/mod/github.com/gogo/status@v1.1.0/status.go:39:2: missing go.sum entry for module providing package google.golang.org/grpc/codes (imported by use); to add:
	go get use
../../go/pkg/mod/github.com/gogo/status@v1.1.0/status.go:40:2: missing go.sum entry for module providing package google.golang.org/grpc/status (imported by github.com/gogo/status); to add:
	go get github.com/gogo/status@v1.1.0

When I updated google.golang.org/protobuf/proto, it also updated the module github.com/golang/protobuf due to the minimum version required in google.golang.org/protobuf. However, github.com/golang/protobuf is not needed to build any package named on the command line, so go get didn't fetch its content or add a hash of its content to go.sum.

The module github.com/golang/protobuf is needed to build the package in the current directory, and since go.sum doesn't contain a hash for its selected version, go build fails.

A possible solution is that if go get changes the selected version of a module and there's a hash for the previous version in go.sum, go get should download the content of the newly selected version and add the hash to go.sum, possibly removing the hash for the previous version. This would make go get a bit slower, since it does more work. This won't prevent all errors, since the new version might import packages from other modules we don't have hashes for.

cc @bcmills @matloob

@jayconrod jayconrod added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. modules labels Feb 5, 2021
@jayconrod jayconrod added this to the Go1.17 milestone Feb 5, 2021
@michaelmdresser
Copy link

I have just encountered this problem as well. Running with Go 1.15 is temporarily fixing my issue, but I would appreciate guidance so that I can use Go 1.16 without trying to mess around with my go.sum.

@jayconrod
Copy link
Contributor Author

For anyone running into this issue in Go 1.16, you can use go get on a package (or set of packages) in the main module. It will add requirements and sums for anything needed by those packages. In my example above, that would be:

go get -t .

go mod tidy will also add requirements sums needed by all packages in the main module. It will also remove requirements and sums that aren't needed anymore.

@navytux
Copy link
Contributor

navytux commented Mar 2, 2021

Unfortunately I'm also hitting this error and neither go get -t . nor go get ... that go1.16 suggests to do change anything and as the result: with go1.16 build fails and cannot be fixed at all, while with go1.15 build succeds ok. In other words it is go1.16 regression that cannot be workerd around.

kirr@deco:~/tmp/trashme$ git clone https://lab.nexedi.com/kirr/wendelin.core
Клонирование в «wendelin.core»…
warning: переадресация на https://lab.nexedi.com/kirr/wendelin.core.git/
remote: Enumerating objects: 15831, done.
remote: Counting objects: 100% (15831/15831), done.
remote: Compressing objects: 100% (4249/4249), done.
remote: Total 15831 (delta 11537), reused 15632 (delta 11373)
Получение объектов: 100% (15831/15831), 7.04 MiB | 2.96 MiB/s, готово.
Определение изменений: 100% (11537/11537), готово.

kirr@deco:~/tmp/trashme$ cd wendelin.core/

kirr@deco:~/tmp/trashme/wendelin.core$ git checkout 481ffb89e78d310f9bdc79f14771494215d0d55a
Note: switching to '481ffb89e78d310f9bdc79f14771494215d0d55a'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD сейчас на 481ffb8 X wcfs: v↑ * (checkpoint)

kirr@deco:~/tmp/trashme/wendelin.core$ cd wcfs/

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.16 build
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:28:2: missing go.sum entry for module providing package crawshaw.io/sqlite (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:29:2: missing go.sum entry for module providing package crawshaw.io/sqlite/sqlitex (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/neo/go@v0.0.0-20210302031331-91085e39ea86/neo/neonet/misc.go:27:2: missing go.sum entry for module providing package github.com/philhofer/fwd (imported by lab.nexedi.com/kirr/neo/go/neo/neonet); to add:
        go get lab.nexedi.com/kirr/neo/go/neo/neonet@v0.0.0-20210302031331-91085e39ea86

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.16 get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.16 get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ git st
HEAD отделён на 481ffb8
нечего коммитить, нет изменений в рабочем каталоге

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.16 get -v lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.16 build
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:28:2: missing go.sum entry for module providing package crawshaw.io/sqlite (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:29:2: missing go.sum entry for module providing package crawshaw.io/sqlite/sqlitex (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/neo/go@v0.0.0-20210302031331-91085e39ea86/neo/neonet/misc.go:27:2: missing go.sum entry for module providing package github.com/philhofer/fwd (imported by lab.nexedi.com/kirr/neo/go/neo/neonet); to add:
        go get lab.nexedi.com/kirr/neo/go/neo/neonet@v0.0.0-20210302031331-91085e39ea86

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.15 build

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.15 get -v lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
go: found lab.nexedi.com/kirr/go123/xnet/lonet in lab.nexedi.com/kirr/go123 v0.0.0-20210302025843-863c4602a230

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ git st
HEAD отделён на 481ffb8
нечего коммитить, нет изменений в рабочем каталоге

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.16 build
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:28:2: missing go.sum entry for module providing package crawshaw.io/sqlite (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:29:2: missing go.sum entry for module providing package crawshaw.io/sqlite/sqlitex (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/neo/go@v0.0.0-20210302031331-91085e39ea86/neo/neonet/misc.go:27:2: missing go.sum entry for module providing package github.com/philhofer/fwd (imported by lab.nexedi.com/kirr/neo/go/neo/neonet); to add:
        go get lab.nexedi.com/kirr/neo/go/neo/neonet@v0.0.0-20210302031331-91085e39ea86

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ GO111MODULE=on go1.15 build

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.16 build
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:28:2: missing go.sum entry for module providing package crawshaw.io/sqlite (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:29:2: missing go.sum entry for module providing package crawshaw.io/sqlite/sqlitex (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/neo/go@v0.0.0-20210302031331-91085e39ea86/neo/neonet/misc.go:27:2: missing go.sum entry for module providing package github.com/philhofer/fwd (imported by lab.nexedi.com/kirr/neo/go/neo/neonet); to add:
        go get lab.nexedi.com/kirr/neo/go/neo/neonet@v0.0.0-20210302031331-91085e39ea86

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ echo $?
1

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.16 get -t .

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ git st
HEAD отделён на 481ffb8
нечего коммитить, нет изменений в рабочем каталоге

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ git diff

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.16 build
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:28:2: missing go.sum entry for module providing package crawshaw.io/sqlite (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:29:2: missing go.sum entry for module providing package crawshaw.io/sqlite/sqlitex (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
/home/kirr/go/pkg/mod/lab.nexedi.com/kirr/neo/go@v0.0.0-20210302031331-91085e39ea86/neo/neonet/misc.go:27:2: missing go.sum entry for module providing package github.com/philhofer/fwd (imported by lab.nexedi.com/kirr/neo/go/neo/neonet); to add:
        go get lab.nexedi.com/kirr/neo/go/neo/neonet@v0.0.0-20210302031331-91085e39ea86

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.15 version
go version go1.15.8 linux/amd64   # built from go1.15.8-6-gf75ab2d5a6

kirr@deco:~/tmp/trashme/wendelin.core/wcfs$ go1.16 version
go version go1.16 linux/amd64     # built from go1.16-12-g2b7243a62f

@navytux
Copy link
Contributor

navytux commented Mar 2, 2021

In other words it is go1.16 regression that cannot be workerd around.

So please change milestone to go1.16.1; thanks.

@davecheney
Copy link
Contributor

@jayconrod here is a sample application, created with Go 1.15 that will not build with Go 1.16

https://github.com/davecheney/issue44129

(~/devel/issue44129) % go1.15 build
(~/devel/issue44129) % go1.16 build
../../pkg/mod/gopkg.in/alecthomas/kingpin.v2@v2.2.6/usage.go:10:2: missing go.sum entry for module providing package github.com/alecthomas/template; try 'go mod tidy' to add it
(~/devel/issue44129) % go mod tidy
(~/devel/issue44129) % go1.16 build
../../pkg/mod/gopkg.in/alecthomas/kingpin.v2@v2.2.6/usage.go:10:2: missing go.sum entry for module providing package github.com/alecthomas/template; try 'go mod tidy' to add it
(~/devel/issue44129) % go get -t
(~/devel/issue44129) % go1.16 build
../../pkg/mod/gopkg.in/alecthomas/kingpin.v2@v2.2.6/usage.go:10:2: missing go.sum entry for module providing package github.com/alecthomas/template; try 'go mod tidy' to add it

@billinghamj
Copy link

billinghamj commented Mar 2, 2021

I'm also finding that this seems to be unresolvable. Have got two examples of it here:

Tried go get ./..., go get -t ./..., running the specific items it suggests, running go mod tidy - none result in it changing any files to cause go mod download to start working again.

The only thing which seems to work is setting -mod=mod on the build - so I'm pretty sure this is a bug in some form or another

@michaelmdresser
Copy link

I was able to resolve my project's issue with go mod tidy.

→ go version 
go version go1.16 linux/amd64

It sounds like not everyone is able to resolve it like this, but I wanted to give an update.

@bcmills
Copy link
Contributor

bcmills commented Mar 2, 2021

@davecheney, thanks for the self-contained repro. I note that go1.15.8 mod tidy in that repro produces diffs, so the fact that the build fails due to an incomplete go.sum is not unexpected given #40728. You should be able to fix that case by running go mod tidy (with either Go 1.15.8 or Go 1.16), or by setting -mod=mod explicitly (perhaps as GOFLAGS=-mod=mod).

issue44129$ go1.15.8 mod tidy

issue44129$ git diff
diff --git i/go.mod w/go.mod
index 8bffa88..1d169f5 100644
--- i/go.mod
+++ w/go.mod
@@ -5,5 +5,5 @@ go 1.15
 require (
        github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 // indirect
        github.com/alecthomas/units v0.0.0-20210208195552-ff826a37aa15 // indirect
-       gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
+       gopkg.in/alecthomas/kingpin.v2 v2.2.6
 )
diff --git i/go.sum w/go.sum
index 93459e1..1a5a037 100644
--- i/go.sum
+++ w/go.sum
@@ -1,11 +1,17 @@
+github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 h1:JYp7IbQjafoB+tBA3gMyHYHrpOtNuDiK/uB5uXxq5wM=
 github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/units v0.0.0-20210208195552-ff826a37aa15 h1:AUNCr9CiJuwrRYS3XieqF+Z9B9gNxo/eANAJCF2eiN4=
 github.com/alecthomas/units v0.0.0-20210208195552-ff826a37aa15/go.mod h1:OMCwj8VM1Kc9e19TLln2VL61YJF0x1XFtfdL4JdbSyE=
+github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

issue44129$ go1.16 build

issue44129$

@bcmills
Copy link
Contributor

bcmills commented Mar 2, 2021

@navytux, using your repro steps and a go1.16 obtained using go install golang.org/dl/go1.16, I do see a change in go.sum from go get …/lonet@….

I also see an error from the suggested `go get` command for `…/neonet@…`; that error occurs at build time (it looks like a bug in the package with `GOOS=linux` and `GOARCH=amd64`), but the error can be suppressed by adding the `-d` flag.

I think the latter is probably a bug, but...

[Edit: it does not appear to be a bug. I accidentally ran one command using a gotip build instead, and the package is broken with a Go built from tip.]

It's not at all clear to me why you aren't seeing any changes at all from the suggested go get …/lonet@… command. Are you using a go command installed from the official builds (https://golang.org/dl), or something carrying patches from a third-party distribution?

wendelin.core/wcfs$ git reset --hard
HEAD is now at 481ffb8 X wcfs: v↑ * (checkpoint)

wendelin.core/wcfs$ go1.16 build .
../../.gopath/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:28:2: missing go.sum entry for module providing package crawshaw.io/sqlite (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
../../.gopath/pkg/mod/lab.nexedi.com/kirr/go123@v0.0.0-20210302025843-863c4602a230/xnet/lonet/registry_sqlite.go:29:2: missing go.sum entry for module providing package crawshaw.io/sqlite/sqlitex (imported by lab.nexedi.com/kirr/go123/xnet/lonet); to add:
        go get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230
../../.gopath/pkg/mod/lab.nexedi.com/kirr/neo/go@v0.0.0-20210302031331-91085e39ea86/neo/neonet/misc.go:27:2: missing go.sum entry for module providing package github.com/philhofer/fwd (imported by lab.nexedi.com/kirr/neo/go/neo/neonet); to add:
        go get lab.nexedi.com/kirr/neo/go/neo/neonet@v0.0.0-20210302031331-91085e39ea86

wendelin.core/wcfs$ go1.16 get lab.nexedi.com/kirr/go123/xnet/lonet@v0.0.0-20210302025843-863c4602a230

wendelin.core/wcfs$ git diff
diff --git i/wcfs/go.sum w/wcfs/go.sum
index be8f8a9..140e969 100644
--- i/wcfs/go.sum
+++ w/wcfs/go.sum
@@ -1,6 +1,7 @@
 cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
 cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
 crawshaw.io/iox v0.0.0-20181124134642-c51c3df30797/go.mod h1:sXBiorCo8c46JlQV3oXPKINnZ8mcqnye1EkVkqsectk=
+crawshaw.io/sqlite v0.3.2 h1:N6IzTjkiw9FItHAa0jp+ZKC6tuLzXqAYIv+ccIWos1I=
 crawshaw.io/sqlite v0.3.2/go.mod h1:igAO5JulrQ1DbdZdtVq48mnZUBAPOeFzer7VhDWNtW4=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
 github.com/DataDog/czlib v0.0.0-20160811164712-4bc9a24e37f2 h1:/Xf0UwdF1dlv6cNu0q1TWTQgfSsALgXrdHNtBMj7ybU=

wendelin.core/wcfs$ go1.16 build .
../../.gopath/pkg/mod/lab.nexedi.com/kirr/neo/go@v0.0.0-20210302031331-91085e39ea86/neo/neonet/misc.go:27:2: missing go.sum entry for module providing package github.com/philhofer/fwd (imported by lab.nexedi.com/kirr/neo/go/neo/neonet); to add:
        go get lab.nexedi.com/kirr/neo/go/neo/neonet@v0.0.0-20210302031331-91085e39ea86

wendelin.core/wcfs$ go1.16 get lab.nexedi.com/kirr/neo/go/neo/neonet@v0.0.0-20210302031331-91085e39ea86

wendelin.core/wcfs$ git diff
diff --git i/wcfs/go.sum w/wcfs/go.sum
index be8f8a9..7d7664f 100644
--- i/wcfs/go.sum
+++ w/wcfs/go.sum
@@ -1,6 +1,7 @@
 cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
 cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
 crawshaw.io/iox v0.0.0-20181124134642-c51c3df30797/go.mod h1:sXBiorCo8c46JlQV3oXPKINnZ8mcqnye1EkVkqsectk=
+crawshaw.io/sqlite v0.3.2 h1:N6IzTjkiw9FItHAa0jp+ZKC6tuLzXqAYIv+ccIWos1I=
 crawshaw.io/sqlite v0.3.2/go.mod h1:igAO5JulrQ1DbdZdtVq48mnZUBAPOeFzer7VhDWNtW4=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
 github.com/DataDog/czlib v0.0.0-20160811164712-4bc9a24e37f2 h1:/Xf0UwdF1dlv6cNu0q1TWTQgfSsALgXrdHNtBMj7ybU=
@@ -72,6 +73,7 @@ github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
 github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348/go.mod h1:B69LEHPfb2qLo0BaaOLcbitczOKLWTsrBG9LczfCD4k=
 github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
 github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
+github.com/philhofer/fwd v1.1.1 h1:GdGcTjf5RNAxwS4QLsiMzJYj5KEvPJD3Abr261yRQXQ=
 github.com/philhofer/fwd v1.1.1/go.mod h1:gk3iGcWd9+svBvR0sR+KPcfE+RNWozjowpeBVG3ZVNU=
 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=

wendelin.core/wcfs$ go1.16 build .

wendelin.core/wcfs$

@bcmills
Copy link
Contributor

bcmills commented Mar 2, 2021

@billinghamj, I'm having a hard time seeing the relevance of those links to this issue. Could you post specific steps to reproduce what you're seeing, and/or a link to the specific build logs?

(Filing a new issue might be ideal, because it seems pretty clear to me that there are multiple different failure modes being discussed on this one.)

@bcmills
Copy link
Contributor

bcmills commented Mar 2, 2021

@navytux, I figured out the go get -d discrepancy above: I accidentally ran that particular go get using gotip, but go1.16 build using Go 1.16. I've edited the trace to show 1.16 only, and then I can't reproduce any inconsistency at all.

I think there is something wrong with your build of the go command; I suggest you open a new issue with full go env output and more detail about how you installed it.

@billinghamj
Copy link

billinghamj commented Mar 2, 2021

@bcmills Sure :) not a huge issue for us any more as we've used the -mod=mod workaround, but it's definitely not behaving the same way on 1.16 on my local machine (darwin/arm64 versus linux/amd64 on the build server, not that it should make any diff)

This is the failing build: https://github.com/cuvva/cuvva-public-go/runs/2011625714#step:4:62

And the tree at the time: https://github.com/cuvva/cuvva-public-go/tree/b78beb17ad162d59cdf31d23e37903424513f4f4

Also in my case, on my local machine, installed using the off-the-shelf brew bottle, not from source etc

@seankhliao
Copy link
Member

Seems to be a dependabot issue for not updating go.sum files?

@bcmills
Copy link
Contributor

bcmills commented Mar 2, 2021

@billinghamj, thanks! Looking at the specific error message, and at the contents of cuvva/cuvva-public-go@b78beb1, I think @seankhliao has it right: dependabot is adding a checksum for the updated dependency's go.mod file, but for some reason not also adding the checksum for the source code.

That particular failure mode looks pretty much exactly like #41103 (CL 251880).

So I'm going to guess that your dependabot is using a Go 1.15 build of the go command and also not re-running go mod tidy after updating the dependencies.

@bcmills
Copy link
Contributor

bcmills commented Mar 2, 2021

@billinghamj, I think what you're seeing is exactly dependabot/dependabot-core#2229 (comment).

@billinghamj
Copy link

billinghamj commented Mar 2, 2021

I'm fairly sure that's a separate problem, as the exact same error arises in CI if I run go mod tidy locally and push after dependabot has opened its PR, but don't have a great public example of that at the moment. Will provide a better example if I get another later

@pauldthomson
Copy link

Seeing the same issue since 1.16 and go build -mod=mod seems to be the only way to fix it:

λ go version
go version go1.16 darwin/amd64

λ go build .
/Users/pthomson/go/pkg/mod/github.com/prometheus/client_golang@v1.9.0/prometheus/internal/metric.go:19:2: missing go.sum entry for module providing package github.com/prometheus/client_model/go (imported by github.com/prometheus/client_golang/prometheus/promhttp); to add:
        go get github.com/prometheus/client_golang/prometheus/promhttp@v1.9.0
/Users/pthomson/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/yaml/decoder.go:32:2: missing go.sum entry for module providing package sigs.k8s.io/yaml (imported by github.com/tyro-private/harbourmaster/internal/update); to add:
        go get github.com/tyro-private/harbourmaster/internal/update

λ go get github.com/prometheus/client_golang/prometheus/promhttp@v1.9.0

λ go build .
/Users/pthomson/go/pkg/mod/github.com/prometheus/client_golang@v1.9.0/prometheus/internal/metric.go:19:2: missing go.sum entry for module providing package github.com/prometheus/client_model/go (imported by github.com/prometheus/client_golang/prometheus/promhttp); to add:
        go get github.com/prometheus/client_golang/prometheus/promhttp@v1.9.0
/Users/pthomson/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/yaml/decoder.go:32:2: missing go.sum entry for module providing package sigs.k8s.io/yaml (imported by github.com/tyro-private/harbourmaster/internal/update); to add:
        go get github.com/tyro-private/harbourmaster/internal/update

λ go build -mod=mod

λ

@jayconrod
Copy link
Contributor Author

Hey folks, it seems like we're seeing multiple bugs here.

If go build $pkg is printing the same errors repeatedly, and running the commands recommended by those errors doesn't fix the problem, please open a new issue, reference this issue somewhere in the text, and cc me. We may see a duplicates, but that's better than distinct problems getting combined into one issue.

If possible, please provide specific instructions to reproduce the problem. A minimal example repo is extraordinarily helpful for us.

(This issue is about go get creating new problems after an upgrade; the go get commands from the error messages should still be able to solve those problems though.)

@jayconrod
Copy link
Contributor Author

@pauldthomson That seems like a new bug. Would you mind opening another issue?

If you can narrow this down to a reproducible example, that would be really helpful. In a test module, I tried importing github.com/prometheus/client_golang/prometheus/promhttp, running go mod tidy, then deleting the sum for github.com/prometheus/client_model v0.2.0. That reproduced the first error, but go get github.com/prometheus/client_golang/prometheus/promhttp@v1.9.0 fixed it for me.

@pauldthomson
Copy link

@jayconrod Sure thing. I'll have a go at getting a repro set up and create a new issue. Thanks.

@pauldthomson
Copy link

@jayconrod #44747

@navytux
Copy link
Contributor

navytux commented Mar 3, 2021

@bcmills, thanks for feedback. I'm using Go built from source and it is just a regular build from go1.16-12-g2b7243a62f bootstrapped from go1.4.3-16-g4d5426a570. I've rechecked it once again in a clean environment and the problem disappears. However I've tracked it down to be a problem related to state of ~/go/pkg/mod - probably a stale cache problem. Please see details in #44749.

mergify bot pushed a commit to aws/jsii that referenced this issue Jun 29, 2021
Switch to using `go mod tidy` instead of `go mod download` to work
around golang/go#44129. This caused tests to
fail due to using `go1.16.x`, which is susceptible to this bug (`go1.17`
will have the fix).



---

By submitting this pull request, I confirm that my contribution is made under the terms of the [Apache 2.0 license].

[Apache 2.0 license]: https://www.apache.org/licenses/LICENSE-2.0
@syned13
Copy link

syned13 commented Jul 17, 2021

I'm having this issue with go1.16.4 and newly go1.16.5. go get or go mod tidy didn't fix my issues but go env -w GOFLAGS=-mod=mod actually did

This worked for me! Thank you! Also, @jayconrod reply was very useful!

tstromberg added a commit to packethost/cacher that referenced this issue Aug 13, 2021
tstromberg added a commit to packethost/cacher that referenced this issue Aug 13, 2021
Signed-off-by: Thomas Stromberg <t+github@stromberg.org>
tstromberg added a commit to packethost/cacher that referenced this issue Aug 17, 2021
Signed-off-by: Thomas Stromberg <t+github@stromberg.org>
@Gilwe
Copy link

Gilwe commented Oct 24, 2021

Hey @jayconrod
It still happens on go@1.17 (the -mod=mod workaround is also still working)

@bcmills
Copy link
Contributor

bcmills commented Oct 25, 2021

@Gilwe, this issue is closed and the problem diagnosed from its steps to reproduce was fixed in Go 1.17.

If you are still seeing a problem on a supported release (1.17.2 or 1.16.9 or a build from source at head), please file a new issue with specific steps to reproduce it.

jakubgs added a commit to status-im/infra-role-golang that referenced this issue Nov 17, 2021
https://golang.org/doc/go1.17

This is supposed to fix dependency resolution issues with `gomobile`:
```
missing go.sum entry for module providing package golang.org/x/mobile/cmd/gobind
```
golang/go#44129

Signed-off-by: Jakub Sokołowski <jakub@status.im>
cce added a commit to algorand/indexer that referenced this issue Dec 9, 2021
dshulyak added a commit to dshulyak/go-spacemesh that referenced this issue Jan 11, 2022
bors bot pushed a commit to spacemeshos/go-spacemesh that referenced this issue Jan 14, 2022
## Motivation

The existing approach lacks atomicity, causal durability (e.g. an atx may not be on disk when a ballot is saved), and the durability can't be enforced in general without running every leveldb operation in sync mode. All this problems will result in subtle bugs that are hard to diagnose.

For those 3 requirements, we want to maintain all state in a single db. Moving state to a single leveldb will require us to enforce isolation ourselves (by maintaining separate namespace and manually concatenating keys like we do in some modules), beside that we have to create every single index manually while with sqlite we can just do `CREATE INDEX` and sqlite will do it for us and probably do a better job. 

Another significant benefit is that we can duplicate some state in sql table to avoid loading the whole structure into memory. For instance it will be relevant for atx, which is a large (10kb) and usually, after it was validated, we want to know only the associated smesher and weight of the atx. This would be problematic with leveldb, and would require adding custom index.

related: #2918

## Changes
- general plumbing for core database stuff (db, transaction, migrations) using https://github.com/crawshaw/sqlite that is a relatively simple wrapper around C sqlite 
- tables and for layers, blocks and ballots
- reworked ZeroLayer, it is relevant only for hare_output, which can be in 3 states - nil, empty, non-empty. 
  SetZeroLayer update hare_output to empty state, at which tortoise will vote against all blocks within hdist. 
- updated to golang 1.16 for `embed` module. note that there is a bug with go mode tidy in 1.16 so i had to manually fix go.sum and disable go mod tidy on ci - golang/go#44129

## Test Plan
existing and new uts
bors bot pushed a commit to spacemeshos/go-spacemesh that referenced this issue Jan 14, 2022
## Motivation

The existing approach lacks atomicity, causal durability (e.g. an atx may not be on disk when a ballot is saved), and the durability can't be enforced in general without running every leveldb operation in sync mode. All this problems will result in subtle bugs that are hard to diagnose.

For those 3 requirements, we want to maintain all state in a single db. Moving state to a single leveldb will require us to enforce isolation ourselves (by maintaining separate namespace and manually concatenating keys like we do in some modules), beside that we have to create every single index manually while with sqlite we can just do `CREATE INDEX` and sqlite will do it for us and probably do a better job. 

Another significant benefit is that we can duplicate some state in sql table to avoid loading the whole structure into memory. For instance it will be relevant for atx, which is a large (10kb) and usually, after it was validated, we want to know only the associated smesher and weight of the atx. This would be problematic with leveldb, and would require adding custom index.

related: #2918

## Changes
- general plumbing for core database stuff (db, transaction, migrations) using https://github.com/crawshaw/sqlite that is a relatively simple wrapper around C sqlite 
- tables and for layers, blocks and ballots
- reworked ZeroLayer, it is relevant only for hare_output, which can be in 3 states - nil, empty, non-empty. 
  SetZeroLayer update hare_output to empty state, at which tortoise will vote against all blocks within hdist. 
- updated to golang 1.16 for `embed` module. note that there is a bug with go mode tidy in 1.16 so i had to manually fix go.sum and disable go mod tidy on ci - golang/go#44129

## Test Plan
existing and new uts
praminda pushed a commit to praminda/product-microgateway that referenced this issue Jan 20, 2022
1.15.x causes `build constraints exclude all Go files in...` error
during build when the azure client is upgraded to azure-sdk-for-go

 `-mod=mod` was needed due to golang/go#44129
praminda pushed a commit to praminda/product-microgateway that referenced this issue Jan 20, 2022
1.15.x causes `build constraints exclude all Go files in...` error
during build when the azure client is upgraded to azure-sdk-for-go

 `-mod=mod` was needed due to golang/go#44129
Enigmatrix added a commit to Enigmatrix/planlah that referenced this issue Jul 2, 2022
JothamWong pushed a commit to Enigmatrix/planlah that referenced this issue Jul 9, 2022
@golang golang locked and limited conversation to collaborators Oct 25, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
None yet
Development

Successfully merging a pull request may close this issue.