cmd/go: private repos on hard-coded hosting providers require special configuration #26134

brunetto · 2018-06-29T14:42:54Z

What version of Go are you using (`go version`)?

$ go get -u golang.org/x/vgo
$ vgo version
go version go1.10 darwin/amd64 vgo:2018-02-20.1

Does this issue reproduce with the latest release?

yes (coming from #25590)

What operating system and processor architecture are you using (`go env`)?

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/brunetto.ziosi/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/brunetto.ziosi/Code"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/m3/145jfhdd5qsf66_41wt9p02mhc4hp5/T/go-build518939426=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I ran

vgo get github.com/MY_ORGANIZATION/MY_REPO

where:

I ran the command in a folder with a working golang app and a proper go.mod file -> vgo is working here and I can vgo build the app
the repo is a private repo with SSO enabled I can access (git pull/push, ...)

What did you expect to see?

The package code in MY_REPO or the go module downloaded

What did you see instead?

vgo get github.com/MY_ORGANIZATION/MY_REPO: git ls-remote -q https://github.com/MY_ORGANIZATION/MY_REPO in /Users/brunetto.ziosi/Code/src/mod/cache/vcs/2cb00fa3547080bb301974fe87cf912af55d50f7a6d8a9955e11fed7a20ac6d3: exit status 128:
	remote: Repository not found.
	fatal: repository 'https://github.com/MY_ORGANIZATION/MY_REPO/' not found

The text was updated successfully, but these errors were encountered:

bcmills · 2018-06-29T21:57:55Z

What vgo commit did you build from?

What happens if you attempt to go get the same repository with the ordinary go tool?

Are you using the credential helper?

brunetto · 2018-07-02T14:21:29Z

What vgo commit did you build from?

I don't know, I just updated it before running it... how can I know the commit?

What happens if you attempt to go get the same repository with the ordinary go tool?

The simple go get does some stuff in the background and says nothing, if I run go get -v -u I see the repo itself downloaded and some dependencies downloaded, few others blocked by our firewall (-insecure seems to allow the download for everything).

Are you using the credential helper?

I don't think so, I exchanged the keys with github so my remote is like

$ git remote -v
origin	git@github.com:MY_ORGANIZATION/MY_REPO.git (fetch)
origin	git@github.com:MY_ORGANIZATION/MY_REPO.git (push)

rsc · 2018-07-06T03:38:01Z

The error text shows the command being run. It is one of the few git commands that doesn't actually care about having a git repo to run, so you can just try it in any directory you want:

git ls-remote -q https://github.com/MY_ORGANIZATION/MY_REPO

Does it work?

rsc · 2018-07-06T03:38:30Z

Possible duplicate of #26145 but the error message from git is different (no mention of terminal prompts disabled).

brunetto · 2018-07-06T06:24:25Z

Ok, so

$ git ls-remote -q https://github.com/MY_ORGANIZATION/MY_REPO
remote: Repository not found.
fatal: repository 'https://github.com/MY_ORGANIZATION/MY_REPO/' not found

but

$ git ls-remote git@github.com:MY_ORGANIZATION/MY_REPO.git
3fe3e3f8b07ff40cdf5de8685360715c4c06df9c	HEAD
1fd67741650c052e30bce6d780cd26c273ba0ab2	refs/heads/dev
1498d237221b9e19d55148dded8fafc7316a47bf	refs/heads/evo
3fe3e3f8b07ff40cdf5de8685360715c4c06df9a	refs/heads/master
3e86bf596a1f63631b91cc9cbbc0866b88cb67c3	refs/pull/1/head
14aef40c003sw32ab1402b2471d164c549624e5c	refs/pull/2/head
...

Maybe I am able to reach the repository only via ssh?

rsc · 2018-07-10T16:31:18Z

If you can make the https form work then vgo (and old go get) will be happy. The two possible ways to do that are:

Add to $HOME/.gitconfig:

[url "ssh://git@github.com/MYORGANIZATION/"]
insteadOf = https://github.com/MYORGANIZATION/
Add to $HOME/.netrc:

machine github.com login YOU password APIKEY

where APIKEY is an API key obtained from the GitHub API page with access to private repos.

rsc · 2018-07-10T16:32:55Z

Retitling this to "cmd/go: github private repos require special configuration".
Maybe for Go 1.12 we should think about some way to get this right by default. I'm not sure exactly how.

brunetto · 2018-07-10T20:23:25Z

The first option, adding

[url "ssh://git@github.com/MYORGANIZATION/"]
    insteadOf = https://github.com/MYORGANIZATION/

to $HOME/.gitconfig work like a charm!!
Thanks!

krak3n · 2018-07-24T09:41:19Z

Hi Guys,

I don't know if this is the correct issue to comment on so please forgive me if it's not correct, I ended up here because of an Issue I am having with Gitlab. We have a project structure where we use subgroups for project organisation by client/project/repo. So our package import paths are 3 levels deep rather than the standard 2 you see on Github. This is fine except for an issue with go get. Gitlab has a strict security policy where requests for the go-import meta data is incorrect for private sub groups by design for unauthenticated requests. So for example if I wanted to go get my.gitlab.com/foo/bar/fizz the meta data request would be https://my.gitlab.com/foo/bar/fizz?go-get=1 and the meta data returned would be:

<html><head><meta name="go-import" content="my.gitlab.com/foo/bar git https://my.gitlab.com/foo/bar.git" /></head></html>

Gitlab recently did implement support for a HTTP header to override this behaviour over here: https://gitlab.com/gitlab-org/gitlab-ce/issues/42817 so we can send a Private-Token header to get the correct meta. However I cannot find a way where I can get go get to send that header to resolve the source path.

We could update our import paths to include a .git suffix but that feels a little broken?

I guess what we need is a way for go get to find crednetial information for these sorts of projects, perhaps another file that lives alongside go.mod and go.sub like go.creds or something, which could contain credential information (you wouldn't want to check that into a repo tho). Or perhaps a way to override go-import meta data locally? 🤷‍♂️

myitcv · 2018-08-08T16:16:12Z

To add to the list of options in this space (#26134 (comment)), we should also add use of a keychain/keyring as a third option (which is independent of remote VCS). Instructions vary quite widely between platforms, unsurprisingly:

Mac: https://help.github.com/articles/updating-credentials-from-the-osx-keychain/
Linux: https://wiki.archlinux.org/index.php/GNOME/Keyring (or equivalent)
Windows: ???

I've switched back from using the ssh solution on Linux to using Gnome keyring (even though I don't use Gnome) to great effect... not least because it can be used to unify all credentials.

@FiloSottile are there are any security implications that I'm unaware of with respect to the use of keychain/keyrings for this sort of thing?

bcmills · 2023-01-19T21:16:36Z

Note that this issue only applies to the hard-coded hosting sites.

I have reason to believe that this statement may have been mistaken. I plan to investigate further.

This issue somehow ended up in the Proposals queue in Feb. 2021, but I'm not sure why — I think it may be just a bug.

bcmills · 2024-03-07T19:36:24Z

I've started to suspect that this may just be a problem in the way that we probe for which protocol to use.

It appears that both GitHub and GitLab support SSH by having everyone use the username git, and then treating the public key presented by the user as the “effective user ID”. So probably when we probe for SSH support, in addition to trying ssh with the user's actual username, we should also try ssh with the hard-coded username git.

matloob · 2024-05-16T18:29:55Z

This needs some more thought and investigation

seankhliao · 2024-12-05T18:22:48Z

If you need conditional git includes, using hasconfig:remote is better:

[includeIf "hasconfig:remote.*.url:https://github.com/org/*"]
  path = org.gitconfig

That said, should this be retitled to focus on ssh probing?

seankhliao · 2025-02-21T09:29:29Z

if you want to use gitdir, you'll need to set the GOAUTH variable https://pkg.go.dev/cmd/go#hdr-GOAUTH_environment_variable

gopherbot added this to the vgo milestone Jun 29, 2018

bcmills added WaitingForInfo NeedsInvestigation labels Jun 29, 2018

rsc changed the title ~~x/vgo: Private repository not found~~ cmd/go: github private repos require special configuration Jul 10, 2018

rsc modified the milestones: vgo, Go1.12 Jul 10, 2018

rsc added NeedsDecision and removed NeedsInvestigation WaitingForInfo labels Jul 10, 2018

bcmills mentioned this issue Aug 9, 2018

cmd/go: we're fetching github.com?go-get=1 #26885

Closed

This comment has been minimized.

Sign in to view

rasky mentioned this issue Aug 20, 2018

cmd/go: improve error message if remote repo does not support https #27088

Closed

myitcv added the modules label Sep 7, 2018

myitcv mentioned this issue Sep 7, 2018

cmd/go: custom import path private repos require special configuration #27254

Closed

jkomoros mentioned this issue Sep 9, 2018

Support modules jkomoros/boardgame#662

Open

This comment has been minimized.

Sign in to view

myitcv mentioned this issue Nov 5, 2018

go mod attempts to download packages when it detects a VCS, even if a replace directive would override the package #28258

Closed

bcmills modified the milestones: Go1.12, Go1.13 Nov 13, 2018

This comment has been minimized.

Sign in to view

bcmills mentioned this issue Sep 25, 2020

proposal: cmd/go: extend syntax go.mod to allow overriding fetch protocol #39536

Open

rsc changed the title ~~cmd/go: private repos on hard-coded hosting providers require special configuration~~ propsoal: cmd/go: private repos on hard-coded hosting providers require special configuration Feb 2, 2021

rsc changed the title ~~propsoal: cmd/go: private repos on hard-coded hosting providers require special configuration~~ proposal: cmd/go: private repos on hard-coded hosting providers require special configuration Feb 2, 2021

gopherbot removed the NeedsDecision label Feb 2, 2021

bcmills mentioned this issue Nov 15, 2021

x/wiki,cmd/go: document how to fetch private repos with Yubikey SSH authentication #49515

Open

Freyert mentioned this issue Apr 14, 2022

proposal: cmd/go: allow GOPRIVATE to provide source repository URI #45611

Open

bcmills mentioned this issue May 10, 2022

cmd/go: support for Git LFS via new VCS qualifier #47308

Open

bcmills mentioned this issue May 17, 2022

cmd/go: Permission Denied when downloading a private module with go get #52942

Closed

rsc moved this to Incoming in Proposals Aug 10, 2022

rsc added this to Proposals Aug 10, 2022

bcmills self-assigned this Jan 19, 2023

bcmills added the NeedsInvestigation label Jan 19, 2023

bcmills changed the title ~~proposal: cmd/go: private repos on hard-coded hosting providers require special configuration~~ cmd/go: private repos on hard-coded hosting providers require special configuration Jan 19, 2023

bcmills removed this from Proposals Jan 19, 2023

bcmills mentioned this issue Jan 9, 2024

cmd/go: go mod tidy not using .git/config settings #65041

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: private repos on hard-coded hosting providers require special configuration #26134

cmd/go: private repos on hard-coded hosting providers require special configuration #26134

brunetto commented Jun 29, 2018

bcmills commented Jun 29, 2018 •

edited

Loading

brunetto commented Jul 2, 2018

rsc commented Jul 6, 2018

rsc commented Jul 6, 2018

brunetto commented Jul 6, 2018

rsc commented Jul 10, 2018

rsc commented Jul 10, 2018

brunetto commented Jul 10, 2018

krak3n commented Jul 24, 2018

myitcv commented Aug 8, 2018

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bcmills commented Jan 19, 2023

bcmills commented Mar 7, 2024

This comment has been minimized.

matloob commented May 16, 2024

This comment has been minimized.

seankhliao commented Dec 5, 2024

This comment has been minimized.

seankhliao commented Feb 21, 2025

cmd/go: private repos on hard-coded hosting providers require special configuration #26134

cmd/go: private repos on hard-coded hosting providers require special configuration #26134

Comments

brunetto commented Jun 29, 2018

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

bcmills commented Jun 29, 2018 • edited Loading

brunetto commented Jul 2, 2018

rsc commented Jul 6, 2018

rsc commented Jul 6, 2018

brunetto commented Jul 6, 2018

rsc commented Jul 10, 2018

rsc commented Jul 10, 2018

brunetto commented Jul 10, 2018

krak3n commented Jul 24, 2018

myitcv commented Aug 8, 2018

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bcmills commented Jan 19, 2023

bcmills commented Mar 7, 2024

This comment has been minimized.

matloob commented May 16, 2024

This comment has been minimized.

seankhliao commented Dec 5, 2024

This comment has been minimized.

seankhliao commented Feb 21, 2025

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

bcmills commented Jun 29, 2018 •

edited

Loading