Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go.sum checksum different on Windows vs. Linux #33178

Closed
kjk opened this issue Jul 19, 2019 · 9 comments
Closed

cmd/go: go.sum checksum different on Windows vs. Linux #33178

kjk opened this issue Jul 19, 2019 · 9 comments
Labels
FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@kjk
Copy link

kjk commented Jul 19, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.7 windows/amd64

$go version
go version go1.12.4 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\kjk\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\kjk\go
set GOPROXY=
set GORACE=
set GOROOT=c:\go
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\kjk\src\apps\offdocs\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\kjk\AppData\Local\Temp\go-build015649226=/tmp/go-build -gno-record-gcc-switches

What did you do?

I have a program that imports github.com/alecthomas/chroma. I've created go.sum on Windows (by building a program with go 1.2.7) and checked it in.

I checked it out on Linux (Ubuntu 18.04) and tried to build it with go 1.2.4.

What did you expect to see?

It should work.

What did you see instead?

It failed with:

verifying github.com/alecthomas/chroma@v0.6.5: checksum mismatch
        downloaded: h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU=
        go.sum:     h1:4CN9fmMfN/k6Ppb+djxJqu3BG7WuLkct0dvScOc0w2w=

I fixed it by deleting go.sum but after a build it was indeed different on Linux than on Windows:

-github.com/alecthomas/chroma v0.6.5 h1:4CN9fmMfN/k6Ppb+djxJqu3BG7WuLkct0dvScOc0w2w=
+github.com/alecthomas/chroma v0.6.5 h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU=

Please note I know little about how go.sum works but here's my theory:

  • go build calculates checksum in go.sum as hash of file content
  • go build downloads repositories using git installed on the system
  • git can change newlines depending on core.autocrlf settings, changing the content and therefore changing the hash

On Windows I have:

PS C:\Users\kjk\src\apps\offdocs> git config --list
core.autocrlf=true

On Linux I have:

kjk@kjk-thinkpad25:~/src/onlinetools$ git config --list
core.autocrlf=false

Presumably, we would prefer go.sum not be affected by local value of git's core.autocrlf setting.

I imagine it would be possible for go tool to invoke git in such a way as to over-ride local setting and ensure that core.autocrlf=false when downloading dependent repos locally for module cache.

@dmitshur dmitshur added GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 19, 2019
@dmitshur dmitshur added this to the Go1.14 milestone Jul 19, 2019
@dmitshur dmitshur changed the title go.sum checksum different on Windows vs. Linux cmd/go: go.sum checksum different on Windows vs. Linux Jul 19, 2019
@kjk
Copy link
Author

kjk commented Jul 19, 2019

Or it could be a stale module cache / hash generated with earlier go version.
Apparently there was a change in how hash was calculated.
I failed to create a reproducible by trying to force this. I created:

main.go:

package main

import (
	"fmt"
	_ "github.com/alecthomas/chroma"
)

func main() {
	fmt.Printf("Hello\n")
}

go.mod:

module tst

go 1.12

require github.com/alecthomas/chroma v0.6.5

And a repro.bat:

del /s /q go.sum
git config --unset-all core.autocrlf

git config --add core.autocrlf  true
git config --get core.autocrlf

go clean -modcache
go build .
copy go.sum go.sum1

del /s /q go.sum

git config --unset-all core.autocrlf
git config --add core.autocrlf false
git config --get core.autocrlf

go clean -modcache
go build .
copy go.sum go.sum2

The theory was to re-create go.sum with core.autocrlf=true and core.autocrlf=false but it seems in both cases I get the same:

github.com/alecthomas/chroma v0.6.5 h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU=

But maybe I'm just not reproducing whatever git settings that led to creating:

github.com/alecthomas/chroma v0.6.5 h1:4CN9fmMfN/k6Ppb+djxJqu3BG7WuLkct0dvScOc0w2w=

One way to confirm the "format change" theory would be to rerun this with a version of go before the go.sum format change and see what hash it produces for github.com/alecthomas/chroma.

I don't know when this change happened.

@jayconrod
Copy link
Contributor

There was a change in the way sums were calculated between go1.11.3 and go1.11.4 (we will ensure that never happens again). Is it possible the h1:4CN9... sum or the module it was derived from was from before that? Is there any way to reproduce this after cleaning the cache (go clean -modcache)?

The checksum algorithm explicitly sets core.autocrlf to avoid problems with line endings, so that shouldn't be an issue.

For the record, h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU= is the correct sum (per sum.golang.org).

@jayconrod jayconrod added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jul 19, 2019
@kjk
Copy link
Author

kjk commented Jul 19, 2019

It's possible. I imagine running go build with go <1.11.3 on the sample program in #33178 (comment) and checking the checksum would resolve this for sure.

I don't have a setup with go 1.11.3 thought.

@FiloSottile
Copy link
Contributor

You can get Go 1.11.3 with go get golang.org/dl/go1.11.3 alongside your system Go.

@kjk
Copy link
Author

kjk commented Jul 20, 2019

It's still h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU= with go 1.11 (as best as I can tell).

@bcmills
Copy link
Contributor

bcmills commented Jul 22, 2019

There was an earlier extraction bug (#27153) fixed in Go 1.11.2, and an even earlier one way back in the vgo phase (#26229). Any idea how old the module cache on your Windows machine is?

@bcmills
Copy link
Contributor

bcmills commented Jul 22, 2019

Also note that before Go 1.12, we had no locking or synchronization on the module cache. Is it possible that the cache entry for this module was corrupted due to concurrent go commands?

@kjk
Copy link
Author

kjk commented Jul 22, 2019

Yes, it's possible it comes from corrupted cache.

I've ran go clean -modcache since so no idea when that cache was corrupted.

It's fine to close this bug as I see no way to make progress determining the root cause.

@bcmills
Copy link
Contributor

bcmills commented Jul 22, 2019

Thanks. Do let us know if you see this again going forward, especially if you discover a clue as to the underlying cause.

@bcmills bcmills closed this as completed Jul 22, 2019
@golang golang locked and limited conversation to collaborators Jul 21, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

6 participants