Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: git shallow fetches broken at CL 556358 #66147

Closed
bcmills opened this issue Mar 6, 2024 · 5 comments
Closed

cmd/go: git shallow fetches broken at CL 556358 #66147

bcmills opened this issue Mar 6, 2024 · 5 comments
Assignees
@bcmills
Labels
GoCommand cmd/go modules NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.23

Comments

@bcmills
Copy link
Contributor

bcmills commented Mar 6, 2024

Go version

go version devel go1.23-7986e26a39 Mon Mar 4 18:59:50 2024 +0000 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/usr/local/google/home/bcmills/.cache/go-build'
GOENV='/usr/local/google/home/bcmills/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/usr/local/google/home/bcmills/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/usr/local/google/home/bcmills'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/google/home/bcmills/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/google/home/bcmills/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.23-7986e26a39 Mon Mar 4 18:59:50 2024 +0000'
GODEBUG=''
GCCGO='/usr/bin/gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/usr/local/google/home/bcmills/go/src/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build954033280=/tmp/go-build -gno-record-gcc-switches'

What did you do?

$ go clean -modcache
$ GOPROXY=direct go mod download -json gerrit.wikimedia.org/r/mediawiki@v0.0.0-20240202145822-67da0cbcfdf7

What did you see happen?

$ GOPROXY=direct go mod download -x -json gerrit.wikimedia.org/r/mediawiki@v0.0.0-20240202145822-67da0cbcfdf7
# get https://gerrit.wikimedia.org/r/mediawiki?go-get=1
# get https://gerrit.wikimedia.org/r/mediawiki?go-get=1: 200 OK (0.066s)
mkdir -p /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs # git3 https://gerrit.wikimedia.org/r/mediawiki
# lock /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79.lock
mkdir -p /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79 # git3 https://gerrit.wikimedia.org/r/mediawiki
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git init --bare
0.010s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git init --bare
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git remote add origin -- https://gerrit.wikimedia.org/r/mediawiki
0.006s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git remote add origin -- https://gerrit.wikimedia.org/r/mediawiki
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 67da0cbcfdf7 --
0.007s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 67da0cbcfdf7 --
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git ls-remote -q origin
0.116s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git ls-remote -q origin
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git tag -l
0.005s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git tag -l
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git fetch -f -c protocol.version=2 --depth=1 origin 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e:refs/dummy
0.006s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git fetch -f -c protocol.version=2 --depth=1 origin 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e:refs/dummy
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
0.122s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e --
0.006s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e --
{
        "Path": "gerrit.wikimedia.org/r/mediawiki",
        "Version": "v0.0.0-20240202145822-67da0cbcfdf7",
        "Error": "gerrit.wikimedia.org/r/mediawiki@v0.0.0-20240202145822-67da0cbcfdf7: invalid version: unknown revision 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e",
        "Origin": {
                "VCS": "git",
                "URL": "https://gerrit.wikimedia.org/r/mediawiki",
                "Ref": "67da0cbcfdf7f6e1c40e9d145df77a088fb6231e",
                "RepoSum": "r1:sSchJC9s3IXQjz9XqT6meZ2hU1YmTimbSLTEdXEHdYQ="
        }
}

What did you expect to see?

Successful fetch, as with previous go versions.

It appears that I put the arguments in the wrong order in https://go.dev/cl/556358, inadventently breaking shallow git fetches. That went undiagnosed, because it fixed the symptom I was testing for (#56881) and we didn't have any tests specifically probing for shallowness of git repos in the module cache.

(CC @suzmue @matloob @samthanawalla)
@bcmills bcmills added NeedsFix The path to resolution is known, but the work has not been done. GoCommand cmd/go modules labels Mar 6, 2024
@bcmills bcmills self-assigned this Mar 6, 2024
@bcmills bcmills added this to the Go1.23 milestone Mar 6, 2024
@bcmills bcmills changed the title import/path: issue title cmd/go: git shallow fetches broken at CL 556358 Mar 6, 2024
@bcmills
Copy link
Contributor Author

bcmills commented Mar 6, 2024

Notably, following the commands in that -x trace:

/tmp/mediawiki$ git fetch -f -c protocol.version=2 --depth=1 origin 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e:refs/dummy
error: unknown switch `c'

@gopherbot
Copy link

Change https://go.dev/cl/569422 mentions this issue: cmd/go/internal/modfetch: pass "-c" arguments to git before subcommands

@bcmills
Copy link
Contributor Author

bcmills commented Mar 11, 2024

@gopherbot, please backport to Go 1.22. This was an accidental regression in https://go.dev/cl/556358, and may cause increased client and server load when fetching modules directly from git repositories.

@gopherbot gopherbot mentioned this issue Mar 11, 2024
Closed
@gopherbot
Copy link

Backport issue(s) opened: #66256 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@gopherbot
Copy link

Change https://go.dev/cl/570737 mentions this issue: cmd/go/internal/modfetch: pass "-c" arguments to git before subcommands

gopherbot pushed a commit that referenced this issue Mar 27, 2024
@thanm
[release-branch.go1.22] cmd/go/internal/modfetch: pass "-c" arguments…
0c53f93 
… to git before subcommands

I accidentally transposed the arguments in CL 556358, causing the
shallow 'git fetch' attempt to always fail. That didn't break any
tests because we fall back to a full fetch, which works for nearly all
real Git servers, and we didn't have a test that checked for shallow
fetches.

Tested manually using:
	GOPROXY=direct go mod download -x -json gerrit.wikimedia.org/r/mediawiki@v0.0.0-20240202145822-67da0cbcfdf7

(I'm still thinking about how to add a proper regression test.)

Fixes #66256.
Updates #66147.

Change-Id: I0bb17283bae856f369fd24f29375e507d0999933
Reviewed-on: https://go-review.googlesource.com/c/go/+/569422
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
(cherry picked from commit 2ab9218)
Reviewed-on: https://go-review.googlesource.com/c/go/+/570737
Reviewed-by: Sam Thanawalla <samthanawalla@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bcmills bcmills

Labels
GoCommand cmd/go modules NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.23
Development

No branches or pull requests
2 participants
@bcmills @gopherbot