Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: custom import path private repos require special configuration #27254

Closed
chowey opened this issue Aug 27, 2018 · 14 comments
Closed

cmd/go: custom import path private repos require special configuration #27254

chowey opened this issue Aug 27, 2018 · 14 comments

Comments

@chowey
Copy link

chowey commented Aug 27, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOOS="darwin"

What did you do?

I have a private repo served over ssh, using gitolite.

That private repo is a package I want to use, at e.g. private.repo.net/user/package. It has a folder structure like this:

README.md
a/a.go
b/b.go

My global .gitconfig looks like this, so that go get works:

[url "ssh://git@private.repo.net"]
        insteadOf = https://private.repo.net

In my main package my/package, I have a go file that imports it like this:

import "private.repo.net/user/package/a"

This used to work when using the old $GOPATH. Now I am trying to use modules instead. I run go build my/package which should resolve dependencies and build.

What did you expect to see?

I expected it to build.

What did you see instead?

build my/package: cannot find module for path private.repo.net/user/package/a

I also tried adding it to the go.mod file by running go get private.repo.net/user/package.git. (NOTE: I need to use .git so that it uses ssh.) So it now private.repo.net/user/package.git appears in my go.mod file.

However, when I do go build, it still complains that "build my/package: cannot find module for path private.repo.net/user/package/a".

@chowey
Copy link
Author

chowey commented Aug 27, 2018

It works if I just do:

import "private.repo.net/user/package.git/a"

(emphasis on ".git") Then go build does the right thing, and correctly initializes the go.mod file.

NOTE: I still need to do git config --global url."ssh://git@private.repo.net".insteadOf "https://private.repo.net" to get this to work.

@FiloSottile FiloSottile changed the title go modules over SSH with nested folder structure cmd/go: cannot find module over SSH with nested folder structure Aug 30, 2018
@FiloSottile FiloSottile added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. modules labels Aug 30, 2018
@FiloSottile FiloSottile added this to the Go1.12 milestone Aug 30, 2018
@lopezator
Copy link

lopezator commented Sep 4, 2018

Same error here, import paths like:

my-private-repo.com/user/pkg/a

Simply don't work and prints:

cannot find module for path ....

@myitcv
Copy link
Member

myitcv commented Sep 7, 2018

@chowey the reason the .git solution works is that it helps guide the go tool with information about where the VCS for custom import paths private.repo.net/user/package.git/... resides (namely (https|ssh)://private.repo.net/user/package.git). Note for github.com and other VCS's known to the go tool, this step is not required.

Without this (i.e. with an import path like private.repo.net/user/package/a), the go tool first queries https://private.repo.net/user/package/a?go-get=1 to work out (according to https://tip.golang.org/cmd/go/#hdr-Remote_import_paths) where the VCS for that custom import path lives. I suspect this is where yours and @lopezator's examples are failing.

Hence you need to ensure:

For credentials, either you can provide a $HOME/.netrc:

machine private.repo.net login YOU password APIKEY

or use an OS credential helper (see #26134 (comment)).

Also related to #26232.

Whether you then use https/ssh access to the underlying VCS is entirely up to you.


I think this issue is effectively the custom import path equivalent of #26134. Hence I'm going to retitle it.

@myitcv myitcv changed the title cmd/go: cannot find module over SSH with nested folder structure cmd/go: custom import path private repos require special configuration Sep 7, 2018
@myitcv myitcv removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 7, 2018
@jackwhelpton
Copy link

jackwhelpton commented Sep 7, 2018

I'm not sure if this is germane to this issue or I should open a new one, but I seem to be able to reproduce this despite checking all of the configuration above:

  • even non-nested imports (import "private.repo.net/user/package") are failing with cannot find module for path

  • this behavior is only apparent with GO111MODULE=on; it does not manifest when GO111MODULE=off

  • I have this in my .gitconfig:

    [url "git@private.repo.net"]
    insteadOf = https://private.repo.net

  • I'm running on Windows, and have the credential manager configured. I can pull from the repo fine.

  • I've checked the meta tags from https://private.repo.net/user/package?go-get=1 and they look fine.

@myitcv
Copy link
Member

myitcv commented Sep 7, 2018

@jackwhelpton are you able to share the output of go get -x SOMETHING?

This might well contain sensitive information in your case, so please be careful.

In case you can't share this information publicly, please do say.

@jackwhelpton
Copy link

jackwhelpton commented Sep 7, 2018

Sure. It does look like an authentication problem, but my standard git commands are working OK. Here's what I get from go get -x private.repo.net/user/package

go get -x private.repo.net/user/package
# C:\Projects\go\pkg\mod\cache\vcs\<id> for git2 https://private.repo.net/user/package.git
cd C:\Projects\go\pkg\mod\cache\vcs\<id>; git ls-remote -q https://private.repo.net/user/package.git
0.742s # cd C:\Projects\go\pkg\mod\cache\vcs\<id>; git ls-remote -q https://private.repo.net/user/package.git
go get private.repo.net/user/package: git ls-remote -q https://private.repo.net/user/package.git in C:\Projects\go\pkg\mod\cache\vcs\<id>: exit status 128:
        fatal: 'git@private.repo.net/user/package.git' does not appear to be a git repository
        fatal: Could not read from remote repository.

        Please make sure you have the correct access rights
        and the repository exists.

@myitcv
Copy link
Member

myitcv commented Sep 8, 2018

@jackwhelpton if you run

git ls-remote -q https://private.repo.net/user/package.git

separately, what is the output?

@jackwhelpton
Copy link

jackwhelpton commented Sep 10, 2018

Ah, so this may be a git configuration issue, although it's interacting with modules in a strange way (given my ability to go get successfully when modules are disabled). Here's what I see with that command:

git ls-remote -q https://private.repo.net/user/package.git
fatal: 'git@private.repo.net/user/package.git' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Note that a simple git pull in that directory still succeeds. I've just realized there's one piece of configuration I hadn't explained yet: I'm interacting with that server using a stored SSH key, and PuTTY/Pageant is handling the key exchange.

@myitcv
Copy link
Member

myitcv commented Sep 10, 2018

@jackwhelpton your insteadOf configuration does not look correct:

Try:

[url "ssh://git@private.repo.net"]
insteadOf = https://private.repo.net

instead of (no pun intended):

[url "git@private.repo.net"]
insteadOf = https://private.repo.net

@jackwhelpton
Copy link

That does it! Good spot. Should have guessed it would be something idiotic on my side.

@lopezator
Copy link

Mine started worked today suddenly without changing anything. I guess some package updated something, added it's modules configuration or whatever...

Not sure why I was getting that error though.

@dmitris
Copy link
Contributor

dmitris commented Sep 25, 2018

the reason the .git solution works is that it helps guide the go tool with information about where the VCS for custom import paths private.repo.net/user/package.git/... resides (namely (https|ssh)://private.repo.net/user/package.git)
@myitcv

If the domain contains git - for example, git.private.repo.net or foo.bar.git.company.com, would it not be reasonable for the go tool to make an educated guess that the VCS is git, and act accordingly? I would think that naming the domains for Mercurial or Bazaar with git portion in the domain would be rather pathological and extremely rare (such edge cases could be handled with a fall-back to the regular ?go-get=1 query).

@rsc
Copy link
Contributor

rsc commented Oct 25, 2018

For private.repo.net/user/package to work with go get, two things need to happen:

  1. The go command has to do an HTTPS fetch of that URL and find an appropriate <meta> tag.
  2. The <meta> tag has to specify an accessible git server (or module server but in this case it's git).

I don't see any indication of what meta tag is being used, exactly. If the meta tag says git:// in the git server URL (instead of https://), then you should not need any special gitconfig.

One problem might be the HTTPS fetch needing authentication. That's #26232.

Closing this issue assuming it is either a duplicate of #26232 or else solved by using a git:// URL in the meta tag.

@enugentdt
Copy link

I'm dropping into this after spending a while being extremely frustrated with the modules system.

Say I'm developing five different modules at the same time, all hosted on private GitLab. Since I'm developing them at the same time, and often, I want to work on multiple of them without committing changes, why not just use the local directory I already have cloned? A "central" list of locations of all Go modules somewhere in $GOPATH or $GOROOT would suffice. E.g. a file listing a set of "replace" directives, saying replace gitlab.com/<team>/<project>/<repo> ~/Documents/code/<project>/<repo>. It would save at least me personally one heck of a headache, and it might be a good solution for a problem that frankly was caused by Go backing itself into a corner.

@golang golang locked and limited conversation to collaborators May 10, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

9 participants