Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: get doesn't work with repositories from git.apache.org #10797

Closed
kostya-sh opened this issue May 13, 2015 · 14 comments
Closed

cmd/go: get doesn't work with repositories from git.apache.org #10797

kostya-sh opened this issue May 13, 2015 · 14 comments
Milestone

Comments

@kostya-sh
Copy link
Contributor

Apache maintains read-only Git mirrors at http://git.apache.org/ (See http://www.apache.org/dev/git.html for more details). Only HTTP and Git protocols are supported (i.e. no HTTPS at the moment)

go get fails to clone such repositories. E.g.:

> go get git.apache.org/thrift.git/lib/go/thrift
fatal: repository 'http://git.apache.org/thrift/' not found

This happens because go get extracts incorrect remote repository URL from import path.

E.g. for git.apache.org/thrift.git/lib/go/thrift import path the remote URL used by go is http://git.apache.org/thrift/ while the correct remote URL should be http://git.apache.org/thrift.git (compare git ls-remote http://git.apache.org/thrift.git vs git ls-remote http://git.apache.org/thrift/)

To fix this a special case needs to be added for git.apache.org in cmd/go/vcs.go similar to (github, launchpad, etc).

I am happy to contribute and test a patch if adding support for another special Git repo to go get seems to be reasonable to Go Team.

@ianlancetaylor ianlancetaylor added this to the Go1.5Maybe milestone May 13, 2015
@bradfitz
Copy link
Contributor

Another special case is fine. Send it in. See https://golang.org/doc/contribute.html if you haven't already.

@kostya-sh
Copy link
Contributor Author

@gopherbot
Copy link

CL https://golang.org/cl/10092 mentions this issue.

@nightlyone
Copy link
Contributor

Hmm go get git.apache.org/thrift.git/lib/go/thrift works right now without any changes, because go get prefers the schema git:// over everything else. Is your goal to prefer the http schema instead?

@bradfitz
Copy link
Contributor

In Go 1.5, neither of those will work without --insecure.

@kostya-sh
Copy link
Contributor Author

Sorry for the confusion.

But when I tested this a few days ago git:// scheme was not available (I was getting connection refused errors. And no, I am not behind firewall ;). In fact other users reported similar problems 1 year ago: https://issues.apache.org/jira/browse/THRIFT-2537. So I assumed that apache only provides http:// scheme.

Still I believe to make git.apache.org repos work with Go 1.5 without the need to specify --insecure there are two things required:

  • git.apache.org provides HTTPS
  • my patch applied

If/when apache adds HTTPS support I will test my change request again. But if there will be no plans to add HTTPS support, then I guess this ticket can be closed. Lets wait some time and see what happens.

asfgit pushed a commit to apache/thrift that referenced this issue May 16, 2015
Client: Go
Patch: Konstantin Shaposhnikov <k.shaposhnikov@gmail.com>

This closes #497

git.apache.org/thrift.git/lib/go/thrift is the correct import path as it is used
by default in the generated go code.

Unfortunately "go get" can download this library only using git:// scheme so if a user running "go get" is behind a firewall or Git is down at git.apache.org (which seems to be the case from time to time) then running go get with this path will fail.

I will try to get this fixed in Go 1.5.
Issues to watch:
- https://issues.apache.org/jira/browse/INFRA-9658
- golang/go#10797
@nightlyone
Copy link
Contributor

Ok , I think I found the culprit: All git urls passed to git are actually intended to end in .git. For the git:// schema, git seems to be clever and adds it, when it is missing, so it works by accident.

The fix might be as simple as

diff --git a/src/cmd/go/vcs.go b/src/cmd/go/vcs.go
index 2179000..876fd36 100644
--- a/src/cmd/go/vcs.go
+++ b/src/cmd/go/vcs.go
@@ -104,7 +104,7 @@ var vcsGit = &vcsCmd{
        name: "Git",
        cmd:  "git",

-       createCmd:   "clone {repo} {dir}",
+       createCmd:   "clone {repo}.git {dir}",
        downloadCmd: "pull --ff-only",

        tagCmd: []tagCmd{
@@ -123,7 +123,7 @@ var vcsGit = &vcsCmd{
        tagSyncDefault: "",

        scheme:     []string{"git", "https", "http", "git+ssh"},
-       pingCmd:    "ls-remote {scheme}://{repo}",
+       pingCmd:    "ls-remote {scheme}://{repo}.git",
        remoteRepo: gitRemoteRepo,
 }

But I'm adding tests for repoRootForImportPathStatic now, to avoid regressions. I will add also maximum coverage to it, to avoid further breakage there.

Stay tuned...

@nightlyone
Copy link
Contributor

Ok, to actually test this, I would need to write some mocking around the execs and fake their results.
The returned repoRoot structure in the current test setup stays the same :-/
So the current tests work the same before and afterwards.

Actually the difference between the createCmd and pingCmd is pretty obvious and lead to same or better results.

The amount of code added would be a few lines and will check command construction (unit testing), not command execution (integration testing).
It might look like this: https://github.com/Jimdo/periodicnoise/blob/master/cmd/pn/monitor.go#L102-L146

But I am not really comfortable adding so much (test) code that late in the release cycle and I am also unsure with what repositories the generic repository stuff handling has been tested. It would be the most affected.

Added https://golang.org/cl/10171 for it.

@bradfitz Please advise how to move forward here.

@gopherbot
Copy link

CL https://golang.org/cl/10171 mentions this issue.

@kostya-sh
Copy link
Contributor Author

@nightlyone

As far as I am aware ".git" suffix is just a convention and not a requirement for Git URLs.

When using git:// scheme in case if git://repo exists (and git://repo.git doesn't):

  • Git won't attempt to read from git://repo.git
  • Git will fail to read from git://repo.git URL.

So basically if someone sets up a repository without ".git" suffix "go get" with the proposed change won't work.

@bradfitz bradfitz modified the milestones: Unplanned, Go1.5Maybe May 18, 2015
@nightlyone
Copy link
Contributor

Ok, @kostya-sh ist right. I asked for feedback on this issue at https://plus.google.com/+IngoOeser/posts/NgF42yKTQyg and while many think it is ok, one answer indicates a different story. So I will to abandon the CL.

@kostya-sh
Copy link
Contributor Author

https://issues.apache.org/jira/browse/INFRA-9658 has been fixed and git.apache.org now supports both git:// and https:// access schemes.

When #9637 is fixed (is it going to happen in Go 1.5 btw?) then I will update my CL accordingly.

@bradfitz
Copy link
Contributor

@adg, can you look at this now? You touched cmd/go last. (in particular, the https stuff)

@kostya-sh
Copy link
Contributor Author

@bradfitz, is there any chance this still can go in Go 1.5?

Without this change packages from git.apache.org repositories that used to work with Go 1.4 will only work with Go 1.5 if -insecure option is specified. The change is quite small and it would be nice to have it in the upcoming Go release especially that git.apache.org support HTTPS now.

@rsc rsc closed this as completed in 428ed1e Jul 22, 2015
@mikioh mikioh modified the milestones: Go1.5, Unplanned Jul 23, 2015
@golang golang locked and limited conversation to collaborators Aug 5, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

6 participants