cmd/go: GODEBUG=gocacheverify=1 panic (internal cache error: cache verify failed) #35412

jrick · 2019-11-07T06:21:26Z

What version of Go are you using (`go version`)?

$ go version
go version go1.13.4 openbsd/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jrick/.cache/go-build"
GOENV="/home/jrick/.config/go/env"
GOEXE=""
GOFLAGS="-tags=netgo -ldflags=-extldflags=-static"
GOHOSTARCH="amd64"
GOHOSTOS="openbsd"
GONOPROXY=""
GONOSUMDB=""
GOOS="openbsd"
GOPATH="/home/jrick/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/jrick/src/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/jrick/src/go/pkg/tool/openbsd_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/jrick/src/release/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"

What did you do?

Working on this release build tool: https://github.com/jrick/release

I noticed that sometimes the tool would reliably reproduce different binaries than others would create on a fresh build host environment. I figured this was due to something stale in the cache being used. After a go clean -cache, I was able to reproduce the same bins as the clean environment again, but decided to look closer at possible cache issues.

Running the release tool with GODEBUG=gocacheverify=1, I found that it will always fail during either the dcrwallet or promptsecret build (whichever comes later) due to a failed cache sanity check. Full output below.

You'll probably want to test with this branch which strips the tool down to only build the dcrwallet and promptsecret executables cross-compiled for linux/amd64 to speed up the repro case: https://github.com/jrick/release/tree/gocacheverify

What did you expect to see?

No panic, reproducible builds from any host environment.

What did you see instead?

$ GODEBUG=gocacheverify=1 ./release                                                                                                                                                           
2019/11/07 01:13:11 releasing with /home/jrick/bin/go go version go1.13.4 openbsd/amd64
2019/11/07 01:13:11 build: bin/linux-amd64/dcrwallet
2019/11/07 01:13:39 build: bin/linux-amd64/promptsecret
2019/11/07 01:13:45 go 'build' '-trimpath' '-tags' 'safe,netgo' '-o' '../bin/linux-amd64/promptsecret' '-ldflags' '-buildid= -X github.com/decred/dcrd/internal/version.BuildMetadata=release -X github.com/decred/dcrd/internal/version.PreRelease=rc1 -X github.com/decred/dcrwallet/version.BuildMetadata=release -X github.com/decred/dcrwallet/version.PreRelease=rc1 -X github.com/decred/dcrlnd/build.BuildMetadata=release -X github.com/decred/dcrlnd/build.PreRelease=rc1' 'github.com/decred/dcrd/cmd/promptsecret'
panic: go: internal cache error: cache verify failed: id=2328e72adaac958709416ac7830643c8177d098c0806d8eb297b4215d01fff97 changed:<<<
compile
goos linux goarch amd64
import "golang.org/x/crypto/ssh/terminal"
omitdebug false standard false local false prefix ""
trimpath
modinfo ""
compile compile version go1.13.4 [] []
=
file terminal.go UUva8GPKJRAImAxpK7RK
file util.go CUyxuNJ-ZbgfoGFUPbv7
file util_linux.go oP0jcmTiqjpHCfxiFrd0
import bytes e5qbnzG6ohbK-d1sjBSJ
import golang.org/x/sys/unix dIC-IQlwAKl1YtFuI_hj
import io pfeL2wt2gZJ6u-TFhbJx
import strconv K-8_3HjqRnbBb6a-jF4c
import sync gF8XJTTS_EiSVQJkIClA
import unicode/utf8 FOuK11OK8zQO1uv9kHjO

>>>
old: 9a0efbcf9ff2b591e0fff78e01663c24d1365f2116c19d7fb800b5dd8037093a 149862
new: a7307552d77290c16f96b93a91c5c5975c4840375d6a99037eb93b86d5fe4c74 149862

goroutine 214 [running]:
cmd/go/internal/cache.(*Cache).putIndexEntry(0xc000010cc0, 0x8795acda2ae72823, 0xc8430683c76a4109, 0xebd806088c097d17, 0x97ff1fd015427b29, 0x91b5f29fcffb0e9a, 0x243c66018ef7ffe0, 0x7f9dc116215f36d1, 0x3a093780ddb500b8, 0x24966, ...)
        /home/jrick/src/go/src/cmd/go/internal/cache/cache.go:331 +0x85e
cmd/go/internal/cache.(*Cache).put(0xc000010cc0, 0x8795acda2ae72823, 0xc8430683c76a4109, 0xebd806088c097d17, 0x97ff1fd015427b29, 0xb29520, 0xc00048a050, 0x1, 0x0, 0x0, ...)
        /home/jrick/src/go/src/cmd/go/internal/cache/cache.go:400 +0x2eb
cmd/go/internal/cache.(*Cache).Put(...)
        /home/jrick/src/go/src/cmd/go/internal/cache/cache.go:370
cmd/go/internal/work.(*Builder).updateBuildID(0xc000023860, 0xc0007adcc0, 0xc000348210, 0x23, 0x1, 0x0, 0x0)
        /home/jrick/src/go/src/cmd/go/internal/work/buildid.go:702 +0x548
cmd/go/internal/work.(*Builder).build(0xc000023860, 0xc0007adcc0, 0x0, 0x0)
        /home/jrick/src/go/src/cmd/go/internal/work/exec.go:770 +0x1cc3
cmd/go/internal/work.(*Builder).Do.func2(0xc0007adcc0)
        /home/jrick/src/go/src/cmd/go/internal/work/exec.go:117 +0x36d
cmd/go/internal/work.(*Builder).Do.func3(0xc000026810, 0xc000023860, 0xc0006ce7c0)
        /home/jrick/src/go/src/cmd/go/internal/work/exec.go:177 +0x79
created by cmd/go/internal/work.(*Builder).Do
        /home/jrick/src/go/src/cmd/go/internal/work/exec.go:164 +0x3a1
2019/11/07 01:13:45 exit status 2

The text was updated successfully, but these errors were encountered:

jayconrod · 2019-11-13T23:04:19Z

Thanks for reporting this. There does seem to be a reproducibility problem here. I'm not quite sure where it is yet, but I managed to reduce the repro to this small script:

#!/bin/bash

export GO111MODULE=on
export GODEBUG=gocacheverify=1
export CGO_ENABLED=
export GOOS=
export GOARCH=
export GOFLAGS=-trimpath
export GOCACHE=$(pwd)/gocache

test -e go.mod || go mod init example.com/m
mkdir -p gocache
go clean -cache

go get -d golang.org/x/crypto@v0.0.0-20190611184440-5c40567a22f8
go build golang.org/x/crypto/ssh/terminal
go get -d golang.org/x/crypto@v0.0.0-20190820162420-60c769a6c586
go build golang.org/x/crypto/ssh/terminal

I reproduced this on darwin_amd64 with go1.13.4 and master.

The source files in golang.org/x/crypto/ssh/terminal are identical between the two versions (obviously they would need to be for a cache conflict). The compile commands appear identical except for directory names and -trimpath arguments, but that shouldn't cause any difference in the output. There's no cgo or assembly in these files. There aren't any obviously wrong strings in the output files (username, home directory, temp directory).

I'll dig into the output files to see what the actual difference is and what might be causing it.

jayconrod · 2019-11-13T23:15:07Z

The compiler command with -trimpath looks like this:

/Users/jayconrod/Code/go/pkg/tool/darwin_amd64/compile -o $WORK/b033/_pkg_.a -trimpath "$WORK/b033=>;/Users/jayconrod/go/pkg/mod/golang.org/x/crypto@v0.0.0-20190611184440-5c40567a22f8=>golang.org/x/crypto@v0.0.0-20190611184440-5c40567a22f8" -p golang.org/x/crypto/ssh/terminal -complete -buildid WIon5yNhRc35yOh8djFA/WIon5yNhRc35yOh8djFA -D "" -importcfg $WORK/b033/importcfg -pack -c=4 ./terminal.go ./util.go ./util_linux.go

So we're replacing the module source directory with the module path and version. Those end up in debug info in the compiled .a file. But the module path and version don't appear in the cache key. That's the problem.

gopherbot · 2019-11-14T21:38:13Z

Change https://golang.org/cl/207317 mentions this issue: cmd/go: include module path and version in cache key with -trimpath

ianlancetaylor changed the title ~~GODEBUG=gocacheverify=1 panic (internal cache error: cache verify failed)~~ cmd/go: GODEBUG=gocacheverify=1 panic (internal cache error: cache verify failed) Nov 7, 2019

ianlancetaylor added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Nov 7, 2019

ianlancetaylor added this to the Go1.14 milestone Nov 7, 2019

jayconrod self-assigned this Nov 13, 2019

jayconrod added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 13, 2019

gopherbot closed this as completed in 4de3c7d Nov 14, 2019

golang locked and limited conversation to collaborators Nov 13, 2020

gopherbot added the FrozenDueToAge label Nov 13, 2020

rsc unassigned jayconrod Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: GODEBUG=gocacheverify=1 panic (internal cache error: cache verify failed) #35412

cmd/go: GODEBUG=gocacheverify=1 panic (internal cache error: cache verify failed) #35412

jrick commented Nov 7, 2019 •

edited

jayconrod commented Nov 13, 2019 •

edited

jayconrod commented Nov 13, 2019

gopherbot commented Nov 14, 2019

cmd/go: GODEBUG=gocacheverify=1 panic (internal cache error: cache verify failed) #35412

cmd/go: GODEBUG=gocacheverify=1 panic (internal cache error: cache verify failed) #35412

Comments

jrick commented Nov 7, 2019 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

jayconrod commented Nov 13, 2019 • edited

jayconrod commented Nov 13, 2019

gopherbot commented Nov 14, 2019

jrick commented Nov 7, 2019 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

jayconrod commented Nov 13, 2019 •

edited