Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go get fails with private bitbucket repositories #27344

Closed
savaki opened this issue Aug 29, 2018 · 19 comments
Closed

cmd/go: go get fails with private bitbucket repositories #27344

savaki opened this issue Aug 29, 2018 · 19 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@savaki
Copy link

savaki commented Aug 29, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOOS="linux"
GOARCH="amd64"

What did you do?

I attempted to go get from a private bitbucket repository.

  1. add the following to ~/.gitconfig
[url "git@bitbucket.org:"]
	insteadOf = https://bitbucket.org/
  1. add the following to ~/.ssh/config
Host bitbucket.org
  HostName bitbucket.org
  User git
  IdentityFile ~/.ssh/id_bitbucket
  StrictHostKeyChecking no
  1. ensure that a valid ssh key exists for bitbucket, ~/.ssh/id_bitbucket

  2. use go get on a private bitbucket repo with modules enabled

What did you expect to see?

I expected the private repository to be pulled to the appropriate ${GOPATH}/pkg/mod
directory.

What did you see instead?

go: bitbucket.org/myprivaterepo/projectname@v0.0.0-20180614180234-a79b05b2ef79: https://api.bitbucket.org/2.0/repositories/myprivaterepo/projectname?fields=scm: 403 Forbidden

In the golang-nuts group, Russ had previous posted that this was an issue and could be resolved by hand. With the advent of modules, manually checking out the correct version and putting it into the correct location becomes way more problematic.

@savaki savaki changed the title go get fails with private bitbucket repositories go get fails with private bitbucket repositories (modules enabled) Aug 29, 2018
@FiloSottile FiloSottile changed the title go get fails with private bitbucket repositories (modules enabled) cmd/go: go get fails with private bitbucket repositories Aug 30, 2018
@FiloSottile FiloSottile added modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Aug 30, 2018
@FiloSottile FiloSottile added this to the Go1.12 milestone Aug 30, 2018
@guybrand
Copy link

I had Similar issues in more than one laptop with Ubunto, MacOs and windows, and instead of blind troubleshooting (or reinstall git that sometimes helped mainly on windows), I resolved/worked around with these (temp/troubleshooting) changes/steps :

  1. edit /usr/local/go/src/cmd/go/internal/get/vcs.go, I added (in bold)
    ...
    func bitbucketVCS(...
    for _, vcs := range []string{"git", "hg"} {
    fmt.Println("root:",root,"vcs",vcs)
  2. still editing /usr/local/go/src/cmd/go/internal/get/vcs.go
    func (v *vcsCmd) runVerboseOnly(dir string, cmd string, keyval ...string) error {
    _, err := v.run1(dir, cmd, keyval, true) //was previously false
    save
  3. terminal -> cd /usr/local/go/src/cmd/go
    sudo go build
  4. (you may want to backup your previous bin/go...)
    sudo mv ./go /usr/local/go/bin/go
  5. cd /go-projects-dir/:
    env GIT_TERMINAL_PROMPT=1 go get

that would:

  1. should you the resolution for each repo (just a helper, in case you added "url" to git config etc) per vcs
  2. would turn the silent bash script that pings bitbucket to interactive mode .
    Once you do that, you can see the actual issue which may vary from wrong ssh key, credential cash missing or invalid etc, and even allow you to interact with the command (e.g. if its a missing PW, you can type the password in)

In other words - this will not SOLVE your problem but has a good success rate of showing where the issue is.

tips:
If you are not actively working using mutiple bitbucket accounts make sure you have config --global git name, email, and url (not as you stated):
git config --global user.name "myBKuser"
git config --global user.email "me@myEmail.com"
git config --global url."https://MyCompany@bitbucket.org/".insteadOf "https://bitbucket.org/"
in one case - the latter resolved the issue for me, adding a credential.helper or alternatively ssh key

and last ... return your original bin/go :)

@guybrand
Copy link

Reason and suggested resolution:
The above behavior is a result of missing global git config/credentials, and going through the code - there can be many other similar issues - unless ALL parts are in place.

When go get command tries to pull a bitbucket repo, it first tries to get the repo details as a public repo, resolving in the "403 Forbidden" reply.

Then it tries to check if a private repo is git or hg (as per issue #5375) , first using git, that will also resolve in 403 - if global git config does not include user.name , user.email.
Yet, this would not be enough without cached credentials on cred manager OR ssh key configured for your bitbucket user.
Last, to make sure the git command goes to the right private repo , git global config should either identify this private repo group/initiator whenever you ping bitbucket, or at least when you ping the specific subdomain, for this you need to either
git config --global url."https://MyCompany@bitbucket.org/".insteadOf "https://bitbucket.org/"
or
git config --global url."https://MyCompany@bitbucket.org/".insteadOf "https://bitbucket.org/MyCompany"

vcs.go main problem is not reporting right statuses that will aid the user to resolve the configuration by himself.
A fix would need to:

  1. Replace the "403 Forbidden" reply that looks like an error (but is actually just a ping for a public repo) in : "cant access public repo 'https://api.bitbucket.org/2.0/repositories/my-repo' - trying to connect to a private repo
  2. check if git global config parameters includes user.name , user.email, if not, check hg connection, if that does not work - echo the user with a friendly error that says : "cant access public repo 'https://api.bitbucket.org/2.0/repositories/my-repo', if this is a private repo, please run:

git config --global user.name "your-bitbucket-user"
git config --global user.email "your-bitbucket-email"
git config --global url."https://your-bitbucket-domain@bitbucket.org/".insteadOf "https://bitbucket.org/domain"

  1. if no ssh key is defined/not defined well, the git command will try to prompt for a git password (go get will not support that, but rather through an error), when this fails, vcs.go is currently discarding the error and treats this as a "general error with pinging the repo", in this case, user error should display a friendly error "in order to allow access to a private repo either your git password should be cached in a credential manager, or an ssh key must be supplied" (perhaps with extra details on how to do that on according to the running OS) and the error from the request to bitbucket displayed alongside the friendly error, for cases like the case above (@savaki says he has set up an ssh key, probably something wrong there, and the ping to bitbucket probably returned some error there, my guess is the last git config I suggested using domain@ would have solved it, displaying a friendly error can save the time for guessing) .

Comments:
all the above "warnings/errors" should be stored during the process, and only display if hg fails as well.
github repo's may have similar issues, but lesser, since no need to determine if the repo is git or hg.

If no comments, I can probably set some of the above logic coded in place (vcs.go).

@bcmills
Copy link
Contributor

bcmills commented Nov 9, 2018

A fix would need to: […]

For (1), we shouldn't print anything that looks like an error until we're sure it's actually an error. If we get all the way through the chain of attempts and it's still failing, then we should display a helpful message.

For (2), that seems too specific (to git and to BitBucket). Perhaps a URL (such as https://golang.org/doc/faq#git_https or a Wiki page) would be more helpful? That could collect configuration steps for various providers and tools without baking it into the go tool itself: that way, if we discover a better way to configure private repos we can publish it without updating the tool.

For (3), we should probably pass through the explicit error from the tool. (That's #25982.)

@guybrand
Copy link

  1. Agreed then, I have a good concept to allow that.
  2. Good idea, I would need access to edit the faq in order to do that, I was given go/wiki edit permission, but I dont see an option to edit the faq, any idea who's authorizing that ?
  3. The git error in this case is quite generic (403 Forbidden) adding some local checks (e.g git.global exists) can give a better indication, however we can concatenate the errors to cover both the tool error and local so the user reading https://golang.org/doc/faq#git_https would go to the right section (e.g. "please read the git.global section on https://golang.org/doc/faq#git_https) - I do agree its specific (for git, not for bitbucket), but its not that we have too many vcs's, and git IS the most common.

@bcmills bcmills modified the milestones: Go1.12, Go1.13 Nov 13, 2018
@dsymonds
Copy link
Contributor

This looks to be a duplicate of #16315.

@030
Copy link

030 commented Apr 17, 2019

@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@mrjrieke
Copy link

mrjrieke commented Aug 7, 2019

Is it possible to have an option to use GOPATH by default and for anything it doesn't find, then vgo the rest? That way, private repo's and others like this one could be set up either manually prior to an automated like build.

@ernsheong
Copy link

ernsheong commented Sep 25, 2019

Anyone with a workaround that actually works? (sorry for empty comment)

This has been rather painful and I think my next stop is to go back to GOPATH and restore vendoring

@guybrand
Copy link

guybrand commented Sep 25, 2019 via email

@ernsheong
Copy link

I downgraded to 1.12.9 and was able to do a go get on a private Bitbucket repo

Something is not working right with regard to this in 1.13

@guybrand
Copy link

In Order to be able to help you (and decide if its the same issue as 27344 or need a new related issue), can you complete in the format of this issue details
(i.e. What version of Go are you using (go version)? ... What operating system and processor architecture are you using (go env)? etc)

@langohr
Copy link

langohr commented Sep 25, 2019

I have a similar issue (I think it is the same)
I have reported it here
#34528 (solved)

@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@harrisa1
Copy link

I downgraded to 1.12.9 and was able to do a go get on a private Bitbucket repo

Something is not working right with regard to this in 1.13

I can confirm that this issue happened for me too, I recently updated to 1.13.5 and it wasn't working, downgraded to 1.12.14 and it worked first try.

@bcmills
Copy link
Contributor

bcmills commented Dec 18, 2019

@harrisa1, @ernsheong: see https://golang.org/wiki/NoPlusOne.

Since the original post here was reported against Go 1.11, a failure that occurs with 1.13.5 but not 1.12.14 is unlikely to be the same root cause. Please open a new issue with steps to reproduce the failure.

@bcmills
Copy link
Contributor

bcmills commented Dec 18, 2019

Duplicate of #25982

@bcmills bcmills marked this as a duplicate of #25982 Dec 18, 2019
@bcmills bcmills closed this as completed Dec 18, 2019
@arulrajnet
Copy link

arulrajnet commented May 28, 2020

The answer for whoever coming here from google search.

To setup private bitbucket cloud

git config --global url."git@bitbucket.org:YOUR_COMPANY".insteadOf "https://bitbucket.org/YOUR_COMPANY"

Make sure you have setup SSH key access in bitbucket https://confluence.atlassian.com/bitbucket/set-up-an-ssh-key-728138079.html . Your private key id_rsa file should be in ${HOME}/.ssh/ folder.

@ivansaid
Copy link

ivansaid commented Jul 9, 2020

The answer for whoever coming here for google search.

To setup private bitbucket cloud

git config --global url."git@bitbucket.org:YOUR_COMPANY".insteadOf "https://bitbucket.org/YOUR_COMPANY"

Make sure you have setup SSH key access in bitbucket https://confluence.atlassian.com/bitbucket/set-up-an-ssh-key-728138079.html . Your private key id_rsa file should be in ${HOME}/.ssh/ folder.

I did that same thing after that did this export GOPRIVATE="bitbucket.org/YOUR_COMPANY" closed the console window, restarted my IDE and it worked, i had tried everything already but i worked for me.

@n-insaidoo
Copy link

n-insaidoo commented Jul 28, 2020

What also works is configuring the ~/.netrc file as such:

machine github.com login $GitHubUserName password $GitHubAccessToken
machine bitbucket.org login $BitBucketUserName password $BitBucketAppPassword

(In this case I also specify credentials to be use for GitHub private repos)

And populating the environment variable GOPRIVATE as appropriate.

@bansalvarun
Copy link

Expose the variable as given below may solve the given problem.
export GOPRIVATE="bitbucket.org/YOUR_COMPANY"

References:
https://community.atlassian.com/t5/Bitbucket-questions/How-do-I-run-go-get-on-a-private-repo/qaq-p/1385040
https://stackoverflow.com/questions/38668444/go-get-private-repo-from-bitbucket

@golang golang locked and limited conversation to collaborators Jan 5, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests