Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: GODEBUG=gocacheverify=1 panic (internal cache error: cache verify failed) #35412

Closed
jrick opened this issue Nov 7, 2019 · 3 comments
Closed

cmd/go: GODEBUG=gocacheverify=1 panic (internal cache error: cache verify failed) #35412

jrick opened this issue Nov 7, 2019 · 3 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone
Go1.14

Comments

@jrick
Copy link
Contributor

jrick commented Nov 7, 2019 •
edited
Loading

What version of Go are you using (go version)?

$ go version
go version go1.13.4 openbsd/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jrick/.cache/go-build"
GOENV="/home/jrick/.config/go/env"
GOEXE=""
GOFLAGS="-tags=netgo -ldflags=-extldflags=-static"
GOHOSTARCH="amd64"
GOHOSTOS="openbsd"
GONOPROXY=""
GONOSUMDB=""
GOOS="openbsd"
GOPATH="/home/jrick/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/jrick/src/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/jrick/src/go/pkg/tool/openbsd_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/jrick/src/release/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"

What did you do?

Working on this release build tool: https://github.com/jrick/release

I noticed that sometimes the tool would reliably reproduce different binaries than others would create on a fresh build host environment. I figured this was due to something stale in the cache being used. After a go clean -cache, I was able to reproduce the same bins as the clean environment again, but decided to look closer at possible cache issues.

Running the release tool with GODEBUG=gocacheverify=1, I found that it will always fail during either the dcrwallet or promptsecret build (whichever comes later) due to a failed cache sanity check. Full output below.

You'll probably want to test with this branch which strips the tool down to only build the dcrwallet and promptsecret executables cross-compiled for linux/amd64 to speed up the repro case: https://github.com/jrick/release/tree/gocacheverify

What did you expect to see?

No panic, reproducible builds from any host environment.

What did you see instead?

$ GODEBUG=gocacheverify=1 ./release                                                                                                                                                           
2019/11/07 01:13:11 releasing with /home/jrick/bin/go go version go1.13.4 openbsd/amd64
2019/11/07 01:13:11 build: bin/linux-amd64/dcrwallet
2019/11/07 01:13:39 build: bin/linux-amd64/promptsecret
2019/11/07 01:13:45 go 'build' '-trimpath' '-tags' 'safe,netgo' '-o' '../bin/linux-amd64/promptsecret' '-ldflags' '-buildid= -X github.com/decred/dcrd/internal/version.BuildMetadata=release -X github.com/decred/dcrd/internal/version.PreRelease=rc1 -X github.com/decred/dcrwallet/version.BuildMetadata=release -X github.com/decred/dcrwallet/version.PreRelease=rc1 -X github.com/decred/dcrlnd/build.BuildMetadata=release -X github.com/decred/dcrlnd/build.PreRelease=rc1' 'github.com/decred/dcrd/cmd/promptsecret'
panic: go: internal cache error: cache verify failed: id=2328e72adaac958709416ac7830643c8177d098c0806d8eb297b4215d01fff97 changed:<<<
compile
goos linux goarch amd64
import "golang.org/x/crypto/ssh/terminal"
omitdebug false standard false local false prefix ""
trimpath
modinfo ""
compile compile version go1.13.4 [] []
=
file terminal.go UUva8GPKJRAImAxpK7RK
file util.go CUyxuNJ-ZbgfoGFUPbv7
file util_linux.go oP0jcmTiqjpHCfxiFrd0
import bytes e5qbnzG6ohbK-d1sjBSJ
import golang.org/x/sys/unix dIC-IQlwAKl1YtFuI_hj
import io pfeL2wt2gZJ6u-TFhbJx
import strconv K-8_3HjqRnbBb6a-jF4c
import sync gF8XJTTS_EiSVQJkIClA
import unicode/utf8 FOuK11OK8zQO1uv9kHjO

>>>
old: 9a0efbcf9ff2b591e0fff78e01663c24d1365f2116c19d7fb800b5dd8037093a 149862
new: a7307552d77290c16f96b93a91c5c5975c4840375d6a99037eb93b86d5fe4c74 149862

goroutine 214 [running]:
cmd/go/internal/cache.(*Cache).putIndexEntry(0xc000010cc0, 0x8795acda2ae72823, 0xc8430683c76a4109, 0xebd806088c097d17, 0x97ff1fd015427b29, 0x91b5f29fcffb0e9a, 0x243c66018ef7ffe0, 0x7f9dc116215f36d1, 0x3a093780ddb500b8, 0x24966, ...)
        /home/jrick/src/go/src/cmd/go/internal/cache/cache.go:331 +0x85e
cmd/go/internal/cache.(*Cache).put(0xc000010cc0, 0x8795acda2ae72823, 0xc8430683c76a4109, 0xebd806088c097d17, 0x97ff1fd015427b29, 0xb29520, 0xc00048a050, 0x1, 0x0, 0x0, ...)
        /home/jrick/src/go/src/cmd/go/internal/cache/cache.go:400 +0x2eb
cmd/go/internal/cache.(*Cache).Put(...)
        /home/jrick/src/go/src/cmd/go/internal/cache/cache.go:370
cmd/go/internal/work.(*Builder).updateBuildID(0xc000023860, 0xc0007adcc0, 0xc000348210, 0x23, 0x1, 0x0, 0x0)
        /home/jrick/src/go/src/cmd/go/internal/work/buildid.go:702 +0x548
cmd/go/internal/work.(*Builder).build(0xc000023860, 0xc0007adcc0, 0x0, 0x0)
        /home/jrick/src/go/src/cmd/go/internal/work/exec.go:770 +0x1cc3
cmd/go/internal/work.(*Builder).Do.func2(0xc0007adcc0)
        /home/jrick/src/go/src/cmd/go/internal/work/exec.go:117 +0x36d
cmd/go/internal/work.(*Builder).Do.func3(0xc000026810, 0xc000023860, 0xc0006ce7c0)
        /home/jrick/src/go/src/cmd/go/internal/work/exec.go:177 +0x79
created by cmd/go/internal/work.(*Builder).Do
        /home/jrick/src/go/src/cmd/go/internal/work/exec.go:164 +0x3a1
2019/11/07 01:13:45 exit status 2
@ianlancetaylor ianlancetaylor changed the title GODEBUG=gocacheverify=1 panic (internal cache error: cache verify failed) cmd/go: GODEBUG=gocacheverify=1 panic (internal cache error: cache verify failed) Nov 7, 2019
@ianlancetaylor ianlancetaylor added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Nov 7, 2019
@ianlancetaylor ianlancetaylor added this to the Go1.14 milestone Nov 7, 2019
@jayconrod jayconrod self-assigned this Nov 13, 2019
@jayconrod
Copy link
Contributor

jayconrod commented Nov 13, 2019 •
edited
Loading

Thanks for reporting this. There does seem to be a reproducibility problem here. I'm not quite sure where it is yet, but I managed to reduce the repro to this small script:

#!/bin/bash

export GO111MODULE=on
export GODEBUG=gocacheverify=1
export CGO_ENABLED=
export GOOS=
export GOARCH=
export GOFLAGS=-trimpath
export GOCACHE=$(pwd)/gocache

test -e go.mod || go mod init example.com/m
mkdir -p gocache
go clean -cache

go get -d golang.org/x/crypto@v0.0.0-20190611184440-5c40567a22f8
go build golang.org/x/crypto/ssh/terminal
go get -d golang.org/x/crypto@v0.0.0-20190820162420-60c769a6c586
go build golang.org/x/crypto/ssh/terminal

I reproduced this on darwin_amd64 with go1.13.4 and master.

The source files in golang.org/x/crypto/ssh/terminal are identical between the two versions (obviously they would need to be for a cache conflict). The compile commands appear identical except for directory names and -trimpath arguments, but that shouldn't cause any difference in the output. There's no cgo or assembly in these files. There aren't any obviously wrong strings in the output files (username, home directory, temp directory).

I'll dig into the output files to see what the actual difference is and what might be causing it.

@jayconrod
Copy link
Contributor

The compiler command with -trimpath looks like this:

/Users/jayconrod/Code/go/pkg/tool/darwin_amd64/compile -o $WORK/b033/_pkg_.a -trimpath "$WORK/b033=>;/Users/jayconrod/go/pkg/mod/golang.org/x/crypto@v0.0.0-20190611184440-5c40567a22f8=>golang.org/x/crypto@v0.0.0-20190611184440-5c40567a22f8" -p golang.org/x/crypto/ssh/terminal -complete -buildid WIon5yNhRc35yOh8djFA/WIon5yNhRc35yOh8djFA -D "" -importcfg $WORK/b033/importcfg -pack -c=4 ./terminal.go ./util.go ./util_linux.go

So we're replacing the module source directory with the module path and version. Those end up in debug info in the compiled .a file. But the module path and version don't appear in the cache key. That's the problem.

@jayconrod jayconrod added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 13, 2019
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/207317 mentions this issue: cmd/go: include module path and version in cache key with -trimpath

@golang golang locked and limited conversation to collaborators Nov 13, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
None yet
Milestone
Go1.14
Development

No branches or pull requests
4 participants
@jayconrod @jrick @ianlancetaylor @gopherbot