cmd/go: 'go build' error from 'git ls-remote' when downloading from private repo #33836

lihaic · 2019-08-26T12:52:44Z

What version of Go are you using (`go version`)?

$ go version
go version go1.13rc1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/lhc/.cache/go-build"
GOENV="/home/lhc/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY="*.bmi"
GONOSUMDB="*.bmi"
GOOS="linux"
GOPATH="/home/lhc/go"
GOPRIVATE="*.bmi"
GOPROXY="https://goproxy.io,direct"
GOROOT="/home/lhc/sdk/go1.13rc1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/lhc/sdk/go1.13rc1/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/lhc/gitlab.bmi/tss/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build750408901=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I set two enviroment variable first.
$ go1.13rc1 env -w GOPROXY=https://goproxy.io,direct
$ go1.13rc1 env -w GOPRIVATE=*gitlab.bmi
Then i tried to build my project.
$ go1.13rc1 build -v
And then it failed, because some packages my project rely on is in a private repo.
The error message is:

# cd .; git ls-remote -- https://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
# cd .; git ls-remote -- git+ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
# cd .; git ls-remote -- ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
go: gitlab.bmi/aaa/logv.git@v1.2.2: reading gitlab.bmi/aaa/logv.git/go.mod at revision v1.2.2: unknown revision v1.2.2

I thought this git ls-remote is trying to get tag info, is this supposed to be git ls-remote --tags or git ls remote

Then i modified the source code /home/lhc/sdk/go1.13rc1/src/cmd/go/internal/get/vcs.go line186
pingCmd: "ls-remote -- {scheme}://{repo}", ---> to pingCmd: "ls-remote {scheme}://{repo}",

And then i tried to run go1.13rc1 build -v, and it succeed.

What did you expect to see?

I hope after i set GOPROXY and GOPRIVATE, i could download package from both public repo and private repo.

What did you see instead?

But i can't run go build correctly.

Question is that is that line wrong?

Thank you for your help.

The text was updated successfully, but these errors were encountered:

aofei · 2019-08-26T15:51:37Z

This seems to work as designed. What you need to know is that go isn't smart enough to handle VCS authentications on its own.

See: Why does "go get" use HTTPS when cloning a repository?

bcmills · 2019-08-26T16:27:14Z

Thanks for the report. This may be related to https://golang.org/cl/181237.

Two questions to help us reproduce the problem:

What happens if you try running those ls-remote commands individually on the command line (in an empty temporary directory)?
What version of the git command are you using (as reported by git version)?
- See also doc/install: define minimum supported VCS versions #26746.

lihaic · 2019-08-27T01:10:37Z

$git version
git version 2.7.4

Try running those git ls-remote command in an empty directory.

$mkdir emptydir; cd emptydir
$git ls-remote -- https://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
$git ls-remote -- git+ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
#git ls-remote -- ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]

Running git ls-remote in an empty directory produces the same result.

$git ls-remote https://gitlab.bmi/aaa/logv
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	HEAD
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	refs/heads/master
1dccbdb4d84588a52a147e9f0e98ce222ba7027a	refs/heads/v1
2741972f4356fd428439411eba51e19639fc4df1	refs/tags/v1.0.0
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	refs/tags/v1.1.0
2c05092bd904db3c5d0f3dfa81059bb0808c317e   refs/tags/v1.2.0
1aeae1413f640f643de9d864f09b36c85d5409ca	refs/tags/v1.2.1
1dccbdb4d84588a52a147e9f0e98ce222ba7027a	refs/tags/v1.2.2
$git ls-remote --tags https://gitlab.bmi/aaa/logv
2741972f4356fd428439411eba51e19639fc4df1	refs/tags/v1.0.0
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	refs/tags/v1.1.0
2c05092bd904db3c5d0f3dfa81059bb0808c317e   refs/tags/v1.2.0
1aeae1413f640f643de9d864f09b36c85d5409ca	refs/tags/v1.2.1
1dccbdb4d84588a52a147e9f0e98ce222ba7027a	refs/tags/v1.2.2

But running git ls-remote without the -- or with --tags get all tag info.

By the way, i order to use private repo, i added ssh git config like below.

$ cat ~/.gitconfig
[user]
	name = lhc
	email = lhcgotobed@163.com
[url "ssh://git@gitlab.bmi/"]
	insteadOf = https://gitlab.bmi/

I think this sholud only avoid go tools using https since my private repo don't have https.
@bcmills Appreciate all you help, thanks.

lihaic · 2019-08-27T01:36:03Z

I am currently using ubuntu 16.04, default git version is 2.7.4.

$sudo add-apt-repository -y ppa:git-core/ppa
$sudo apt-get update
$sudo apt-get install git -y
$git version
git version 2.23.0

I did this to upgrade git to v2.23.0, and then git ls-remote -- worked! And with the origin go1.13rc1 binary, go build works fine with my private repo.
So seems like this problem related to git version.

gopherbot · 2019-08-27T18:22:52Z

Change https://golang.org/cl/191978 mentions this issue: cmd/go/internal/get: remove '--' separator from 'git ls-remote' command

bcmills · 2019-08-27T18:27:30Z

@gopherbot, please backport to Go 1.13.

gopherbot · 2019-08-27T18:27:37Z

Backport issue(s) opened: #33880 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

gopherbot · 2019-08-27T20:50:13Z

Change https://golang.org/cl/191973 mentions this issue: [release-branch.go1.13] cmd/go/internal/get: remove '--' separator from 'git ls-remote' command

…om 'git ls-remote' command 'git ls-remote' started recognizing the '--' separator at some point after 2.7.4, but git defaults to version 2.7.4 on Ubuntu 16.04 LTS, which remains supported by Ubuntu until April 2021. We added '--' tokens to most VCS commands as a defensive measure in CL 181237, but it isn't strictly necessary here because the 'scheme' argument to our template is chosen from a predefined list: we can safely drop it to retain compatibility. Cherry-picked from CL 191978. Updates #33836 Fixes #33880 Change-Id: Ibb53366b95f8029b587e0b7646a439330d759ac7 Reviewed-on: https://go-review.googlesource.com/c/go/+/191973 Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>

'git ls-remote' started recognizing the '--' separator at some point after 2.7.4, but git defaults to version 2.7.4 on Ubuntu 16.04 LTS, which remains supported by Ubuntu until April 2021. We added '--' tokens to most VCS commands as a defensive measure in CL 181237, but it isn't strictly necessary here because the 'scheme' argument to our template is chosen from a predefined list: we can safely drop it to retain compatibility. Fixes golang#33836 Updates golang#26746 Change-Id: Ibb53366b95f8029b587e0b7646a439330d759ac7 Reviewed-on: https://go-review.googlesource.com/c/go/+/191978 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

bcmills added modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Aug 26, 2019

bcmills added this to the Go1.13 milestone Aug 26, 2019

bcmills changed the title ~~go1.13rc1: go build can't download private repo package after set GOPROXY and GOPRIVATE~~ cmd/go: 'go build' error from 'git ls-remote' when downloading from private repo Aug 26, 2019

bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 27, 2019

bcmills mentioned this issue Aug 27, 2019

x/build: add a builder with oldest git supported by cmd/go to catch regressions #26653

Open

bcmills modified the milestones: Go1.13, Go1.14 Aug 27, 2019

gopherbot mentioned this issue Aug 27, 2019

cmd/go: 'go build' error from 'git ls-remote' when downloading from private repo [1.13 backport] #33880

Closed

gopherbot closed this as completed in 0a778cf Aug 27, 2019

golang locked and limited conversation to collaborators Aug 26, 2020

gopherbot added the FrozenDueToAge label Aug 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: 'go build' error from 'git ls-remote' when downloading from private repo #33836

cmd/go: 'go build' error from 'git ls-remote' when downloading from private repo #33836

lihaic commented Aug 26, 2019 •

edited

aofei commented Aug 26, 2019

bcmills commented Aug 26, 2019 •

edited

lihaic commented Aug 27, 2019

lihaic commented Aug 27, 2019

gopherbot commented Aug 27, 2019

bcmills commented Aug 27, 2019

gopherbot commented Aug 27, 2019

gopherbot commented Aug 27, 2019

cmd/go: 'go build' error from 'git ls-remote' when downloading from private repo #33836

cmd/go: 'go build' error from 'git ls-remote' when downloading from private repo #33836

Comments

lihaic commented Aug 26, 2019 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

aofei commented Aug 26, 2019

bcmills commented Aug 26, 2019 • edited

lihaic commented Aug 27, 2019

lihaic commented Aug 27, 2019

gopherbot commented Aug 27, 2019

bcmills commented Aug 27, 2019

gopherbot commented Aug 27, 2019

gopherbot commented Aug 27, 2019

lihaic commented Aug 26, 2019 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

bcmills commented Aug 26, 2019 •

edited