code review 14026043: cmd/go: Re-arranged schemes which go get tries to fetch...

cmd/go: Re-arranged schemes which go get tries to fetch git repositories.

git:// was tried first, but would hang for a long time if pinted at a
http:// for https://-server.

Fixes issue 6489.

[msiebuhr, 2013-09-27 07:55:12]

Hello golang-dev@googlegroups.com (cc: sbhr@sbhr.dk),

I'd like you to review this change to
https://code.google.com/p/go/

[bradfitz, 2013-09-28 16:07:54]

Doesn't this just shift the problem?
 On Sep 27, 2013 6:07 AM, <sbhr@sbhr.dk> wrote:

> Reviewers: golang-dev1,
>
> Message:
> Hello golang-dev@googlegroups.com (cc: sbhr@sbhr.dk),
>
> I'd like you to review this change to
> https://code.google.com/p/go/
>
>
> Description:
> cmd/go: Re-arranged schemes which go get tries to fetch git
> repositories.
>
> git:// was tried first, but would hang for a long time if pinted at a
> http:// for https://-server.
>
> Fixes issue 6489.
>
> Please review this at
https://codereview.appspot.**com/14026043/<https://codereview.appspot.com/140...
>
> Affected files (+1, -1 lines):
>   M src/cmd/go/vcs.go
>
>
> Index: src/cmd/go/vcs.go
> ==============================**==============================**=======
> --- a/src/cmd/go/vcs.go
> +++ b/src/cmd/go/vcs.go
> @@ -104,7 +104,7 @@
>         tagSyncCmd:     "checkout {tag}",
>         tagSyncDefault: "checkout master",
>
> -       scheme:  []string{"git", "https", "http", "git+ssh"},
> +       scheme:  []string{"https", "http", "git", "git+ssh"},
>         pingCmd: "ls-remote {scheme}://{repo}",
>  }
>
>
>
> --
>
> ---You received this message because you are subscribed to the Google
> Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to
golang-dev+unsubscribe@**googlegroups.com<golang-dev%2Bunsubscribe@googlegrou...
> .
> For more options, visit
https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/o...
> .
>

[msiebuhr, 2013-09-28 21:09:41]

Brad &al,

I believe git (when running the git://-proto) backs out gracefully when
pointed at a SSH-server, but having just re-tried with another SSH-server
(where it just hung), I might have just been lucky.

I'll get back to you with some more testing, once I've caught up on sleep :)


On Sat, Sep 28, 2013 at 6:07 PM, Brad Fitzpatrick <bradfitz@golang.org>wrote:

> Doesn't this just shift the problem?
>  On Sep 27, 2013 6:07 AM, <sbhr@sbhr.dk> wrote:
>
>> Reviewers: golang-dev1,
>>
>> Message:
>> Hello golang-dev@googlegroups.com (cc: sbhr@sbhr.dk),
>>
>> I'd like you to review this change to
>> https://code.google.com/p/go/
>>
>>
>> Description:
>> cmd/go: Re-arranged schemes which go get tries to fetch git
>> repositories.
>>
>> git:// was tried first, but would hang for a long time if pinted at a
>> http:// for https://-server.
>>
>> Fixes issue 6489.
>>
>> Please review this at
https://codereview.appspot.**com/14026043/<https://codereview.appspot.com/140...
>>
>> Affected files (+1, -1 lines):
>>   M src/cmd/go/vcs.go
>>
>>
>> Index: src/cmd/go/vcs.go
>> ==============================**==============================**=======
>> --- a/src/cmd/go/vcs.go
>> +++ b/src/cmd/go/vcs.go
>> @@ -104,7 +104,7 @@
>>         tagSyncCmd:     "checkout {tag}",
>>         tagSyncDefault: "checkout master",
>>
>> -       scheme:  []string{"git", "https", "http", "git+ssh"},
>> +       scheme:  []string{"https", "http", "git", "git+ssh"},
>>         pingCmd: "ls-remote {scheme}://{repo}",
>>  }
>>
>>
>>
>> --
>>
>> ---You received this message because you are subscribed to the Google
>> Groups "golang-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to
golang-dev+unsubscribe@**googlegroups.com<golang-dev%2Bunsubscribe@googlegrou...
>> .
>> For more options, visit
https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/o...
>> .
>>
>


-- 
Morten Siebuhr

[msiebuhr, 2013-10-07 09:23:50]

Done a bit more testing on this:

 - https and http backs out gracefully when pointed at wrong
combinations of https, http, ssh and git servers.
 - git seem to back out gracefully from *some* SSH servers, but not
all (and I can't figure out why).
 - git hangs on http, but not https (though the error messages vary wildly).
 - ssh hangs when pointed at https, http and git servers.

For maximum coverage of protocols, the order has to be https → http →
git → ssh+git.

To avoid hangs (when given an explicit port-number), git and git+ssh
should be left out (but that would mean breaking go1 compatibility)


On Sat, Sep 28, 2013 at 11:09 PM, Morten Siebuhr <sbhr@sbhr.dk> wrote:
> Brad &al,
>
> I believe git (when running the git://-proto) backs out gracefully when
> pointed at a SSH-server, but having just re-tried with another SSH-server
> (where it just hung), I might have just been lucky.
>
> I'll get back to you with some more testing, once I've caught up on sleep :)
>
>
> On Sat, Sep 28, 2013 at 6:07 PM, Brad Fitzpatrick <bradfitz@golang.org>
> wrote:
>>
>> Doesn't this just shift the problem?
>>
>> On Sep 27, 2013 6:07 AM, <sbhr@sbhr.dk> wrote:
>>>
>>> Reviewers: golang-dev1,
>>>
>>> Message:
>>> Hello golang-dev@googlegroups.com (cc: sbhr@sbhr.dk),
>>>
>>> I'd like you to review this change to
>>> https://code.google.com/p/go/
>>>
>>>
>>> Description:
>>> cmd/go: Re-arranged schemes which go get tries to fetch git
>>> repositories.
>>>
>>> git:// was tried first, but would hang for a long time if pinted at a
>>> http:// for https://-server.
>>>
>>> Fixes issue 6489.
>>>
>>> Please review this at https://codereview.appspot.com/14026043/
>>>
>>> Affected files (+1, -1 lines):
>>>   M src/cmd/go/vcs.go
>>>
>>>
>>> Index: src/cmd/go/vcs.go
>>> ===================================================================
>>> --- a/src/cmd/go/vcs.go
>>> +++ b/src/cmd/go/vcs.go
>>> @@ -104,7 +104,7 @@
>>>         tagSyncCmd:     "checkout {tag}",
>>>         tagSyncDefault: "checkout master",
>>>
>>> -       scheme:  []string{"git", "https", "http", "git+ssh"},
>>> +       scheme:  []string{"https", "http", "git", "git+ssh"},
>>>         pingCmd: "ls-remote {scheme}://{repo}",
>>>  }
>>>
>>>
>>>
>>> --
>>>
>>> ---You received this message because you are subscribed to the Google
>>> Groups "golang-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to golang-dev+unsubscribe@googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
>
> --
> Morten Siebuhr



-- 
Morten Siebuhr

[josharian, 2013-10-07 15:00:25]

I'm not keen on this.

* This sounds like a git problem, not a go get problem. Have you
inquired upstream about fixing this, or about alternative workarounds?
* It moves http ahead of more secure protocols.
* Dealing with git credentials over https is a bit of a pain, so I'd
personally prefer it not be the first choice protocol.
* Is there a go-import META workaround available?
* Might a more aggressive timeout on the ls-remote exec do the trick instead?

-josh


On Mon, Oct 7, 2013 at 2:23 AM, Morten Siebuhr <sbhr@sbhr.dk> wrote:
> Done a bit more testing on this:
>
>  - https and http backs out gracefully when pointed at wrong
> combinations of https, http, ssh and git servers.
>  - git seem to back out gracefully from *some* SSH servers, but not
> all (and I can't figure out why).
>  - git hangs on http, but not https (though the error messages vary wildly).
>  - ssh hangs when pointed at https, http and git servers.
>
> For maximum coverage of protocols, the order has to be https → http →
> git → ssh+git.
>
> To avoid hangs (when given an explicit port-number), git and git+ssh
> should be left out (but that would mean breaking go1 compatibility)
>
>
> On Sat, Sep 28, 2013 at 11:09 PM, Morten Siebuhr <sbhr@sbhr.dk> wrote:
>> Brad &al,
>>
>> I believe git (when running the git://-proto) backs out gracefully when
>> pointed at a SSH-server, but having just re-tried with another SSH-server
>> (where it just hung), I might have just been lucky.
>>
>> I'll get back to you with some more testing, once I've caught up on sleep :)
>>
>>
>> On Sat, Sep 28, 2013 at 6:07 PM, Brad Fitzpatrick <bradfitz@golang.org>
>> wrote:
>>>
>>> Doesn't this just shift the problem?
>>>
>>> On Sep 27, 2013 6:07 AM, <sbhr@sbhr.dk> wrote:
>>>>
>>>> Reviewers: golang-dev1,
>>>>
>>>> Message:
>>>> Hello golang-dev@googlegroups.com (cc: sbhr@sbhr.dk),
>>>>
>>>> I'd like you to review this change to
>>>> https://code.google.com/p/go/
>>>>
>>>>
>>>> Description:
>>>> cmd/go: Re-arranged schemes which go get tries to fetch git
>>>> repositories.
>>>>
>>>> git:// was tried first, but would hang for a long time if pinted at a
>>>> http:// for https://-server.
>>>>
>>>> Fixes issue 6489.
>>>>
>>>> Please review this at https://codereview.appspot.com/14026043/
>>>>
>>>> Affected files (+1, -1 lines):
>>>>   M src/cmd/go/vcs.go
>>>>
>>>>
>>>> Index: src/cmd/go/vcs.go
>>>> ===================================================================
>>>> --- a/src/cmd/go/vcs.go
>>>> +++ b/src/cmd/go/vcs.go
>>>> @@ -104,7 +104,7 @@
>>>>         tagSyncCmd:     "checkout {tag}",
>>>>         tagSyncDefault: "checkout master",
>>>>
>>>> -       scheme:  []string{"git", "https", "http", "git+ssh"},
>>>> +       scheme:  []string{"https", "http", "git", "git+ssh"},
>>>>         pingCmd: "ls-remote {scheme}://{repo}",
>>>>  }
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> ---You received this message because you are subscribed to the Google
>>>> Groups "golang-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to golang-dev+unsubscribe@googlegroups.com.
>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>>
>> --
>> Morten Siebuhr
>
>
>
> --
> Morten Siebuhr
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
"golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to golang-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[rsc, 2013-10-07 15:01:33]

Not for Go 1.2

[msiebuhr, 2013-10-07 19:27:32]

Hi Josh,

On Mon, Oct 7, 2013 at 4:59 PM, Josh Bleecher Snyder
<josharian@gmail.com> wrote:
> I'm not keen on this.
>
> * This sounds like a git problem, not a go get problem. Have you
> inquired upstream about fixing this, or about alternative workarounds?

I haven't talked to the git-people, as I figured this clearly was a
go-related problem. As to alternative workarounds, I really don't have
any good ideas.

> * It moves http ahead of more secure protocols.

Yes. From a security standpoint, I would probably have gone with
https, git+ssh, http and then git. But that would fall apart when
given a non-standard port number (which makes development a pain).


> * Dealing with git credentials over https is a bit of a pain, so I'd
> personally prefer it not be the first choice protocol.

How so? Doesn't git take care of that?


> * Is there a go-import META workaround available?

I'm trying, but I'm having some difficulty making it work.


> * Might a more aggressive timeout on the ls-remote exec do the trick instead?

git ls-remote doesn't have a timeout built-in (per
http://git-scm.com/docs/git-ls-remote.html), so that would have to be
built in the vcs-wrapper, I guess.

> -josh
>
>
> On Mon, Oct 7, 2013 at 2:23 AM, Morten Siebuhr <sbhr@sbhr.dk> wrote:
>> Done a bit more testing on this:
>>
>>  - https and http backs out gracefully when pointed at wrong
>> combinations of https, http, ssh and git servers.
>>  - git seem to back out gracefully from *some* SSH servers, but not
>> all (and I can't figure out why).
>>  - git hangs on http, but not https (though the error messages vary wildly).
>>  - ssh hangs when pointed at https, http and git servers.
>>
>> For maximum coverage of protocols, the order has to be https → http →
>> git → ssh+git.
>>
>> To avoid hangs (when given an explicit port-number), git and git+ssh
>> should be left out (but that would mean breaking go1 compatibility)
>>
>>
>> On Sat, Sep 28, 2013 at 11:09 PM, Morten Siebuhr <sbhr@sbhr.dk> wrote:
>>> Brad &al,
>>>
>>> I believe git (when running the git://-proto) backs out gracefully when
>>> pointed at a SSH-server, but having just re-tried with another SSH-server
>>> (where it just hung), I might have just been lucky.
>>>
>>> I'll get back to you with some more testing, once I've caught up on sleep
:)
>>>
>>>
>>> On Sat, Sep 28, 2013 at 6:07 PM, Brad Fitzpatrick <bradfitz@golang.org>
>>> wrote:
>>>>
>>>> Doesn't this just shift the problem?
>>>>
>>>> On Sep 27, 2013 6:07 AM, <sbhr@sbhr.dk> wrote:
>>>>>
>>>>> Reviewers: golang-dev1,
>>>>>
>>>>> Message:
>>>>> Hello golang-dev@googlegroups.com (cc: sbhr@sbhr.dk),
>>>>>
>>>>> I'd like you to review this change to
>>>>> https://code.google.com/p/go/
>>>>>
>>>>>
>>>>> Description:
>>>>> cmd/go: Re-arranged schemes which go get tries to fetch git
>>>>> repositories.
>>>>>
>>>>> git:// was tried first, but would hang for a long time if pinted at a
>>>>> http:// for https://-server.
>>>>>
>>>>> Fixes issue 6489.
>>>>>
>>>>> Please review this at https://codereview.appspot.com/14026043/
>>>>>
>>>>> Affected files (+1, -1 lines):
>>>>>   M src/cmd/go/vcs.go
>>>>>
>>>>>
>>>>> Index: src/cmd/go/vcs.go
>>>>> ===================================================================
>>>>> --- a/src/cmd/go/vcs.go
>>>>> +++ b/src/cmd/go/vcs.go
>>>>> @@ -104,7 +104,7 @@
>>>>>         tagSyncCmd:     "checkout {tag}",
>>>>>         tagSyncDefault: "checkout master",
>>>>>
>>>>> -       scheme:  []string{"git", "https", "http", "git+ssh"},
>>>>> +       scheme:  []string{"https", "http", "git", "git+ssh"},
>>>>>         pingCmd: "ls-remote {scheme}://{repo}",
>>>>>  }

-- 
Morten Siebuhr

[josharian, 2013-10-07 21:03:17]

>> * Dealing with git credentials over https is a bit of a pain, so I'd
>> personally prefer it not be the first choice protocol.
>
> How so? Doesn't git take care of that?

No, it requires typing or external help:
http://stackoverflow.com/questions/5343068/is-there-a-way-to-skip-password-ty....
I take the number of question upvotes and the complicated top answer
there as some evidence that https is not the easiest way to go here.
Not a huge deal...but not ideal either.


>> * Is there a go-import META workaround available?
>
> I'm trying, but I'm having some difficulty making it work.

Might be worth asking the golang-nuts list about it. They're all kinds
of helpful, in my experience.


>> * Might a more aggressive timeout on the ls-remote exec do the trick
instead?
>
> git ls-remote doesn't have a timeout built-in (per
> http://git-scm.com/docs/git-ls-remote.html), so that would have to be
> built in the vcs-wrapper, I guess.

I think it would, yeah. Speaking only for myself (and I am not a
member of the core team, just a kibitzer), having a timeout in the
vcs-wrapper seems fairly reasonable anyway.

But as Russ said, none of this matters until Go 1.2 is out the door.

-josh



>> On Mon, Oct 7, 2013 at 2:23 AM, Morten Siebuhr <sbhr@sbhr.dk> wrote:
>>> Done a bit more testing on this:
>>>
>>>  - https and http backs out gracefully when pointed at wrong
>>> combinations of https, http, ssh and git servers.
>>>  - git seem to back out gracefully from *some* SSH servers, but not
>>> all (and I can't figure out why).
>>>  - git hangs on http, but not https (though the error messages vary
wildly).
>>>  - ssh hangs when pointed at https, http and git servers.
>>>
>>> For maximum coverage of protocols, the order has to be https → http →
>>> git → ssh+git.
>>>
>>> To avoid hangs (when given an explicit port-number), git and git+ssh
>>> should be left out (but that would mean breaking go1 compatibility)
>>>
>>>
>>> On Sat, Sep 28, 2013 at 11:09 PM, Morten Siebuhr <sbhr@sbhr.dk> wrote:
>>>> Brad &al,
>>>>
>>>> I believe git (when running the git://-proto) backs out gracefully when
>>>> pointed at a SSH-server, but having just re-tried with another SSH-server
>>>> (where it just hung), I might have just been lucky.
>>>>
>>>> I'll get back to you with some more testing, once I've caught up on sleep
:)
>>>>
>>>>
>>>> On Sat, Sep 28, 2013 at 6:07 PM, Brad Fitzpatrick <bradfitz@golang.org>
>>>> wrote:
>>>>>
>>>>> Doesn't this just shift the problem?
>>>>>
>>>>> On Sep 27, 2013 6:07 AM, <sbhr@sbhr.dk> wrote:
>>>>>>
>>>>>> Reviewers: golang-dev1,
>>>>>>
>>>>>> Message:
>>>>>> Hello golang-dev@googlegroups.com (cc: sbhr@sbhr.dk),
>>>>>>
>>>>>> I'd like you to review this change to
>>>>>> https://code.google.com/p/go/
>>>>>>
>>>>>>
>>>>>> Description:
>>>>>> cmd/go: Re-arranged schemes which go get tries to fetch git
>>>>>> repositories.
>>>>>>
>>>>>> git:// was tried first, but would hang for a long time if pinted at a
>>>>>> http:// for https://-server.
>>>>>>
>>>>>> Fixes issue 6489.
>>>>>>
>>>>>> Please review this at https://codereview.appspot.com/14026043/
>>>>>>
>>>>>> Affected files (+1, -1 lines):
>>>>>>   M src/cmd/go/vcs.go
>>>>>>
>>>>>>
>>>>>> Index: src/cmd/go/vcs.go
>>>>>> ===================================================================
>>>>>> --- a/src/cmd/go/vcs.go
>>>>>> +++ b/src/cmd/go/vcs.go
>>>>>> @@ -104,7 +104,7 @@
>>>>>>         tagSyncCmd:     "checkout {tag}",
>>>>>>         tagSyncDefault: "checkout master",
>>>>>>
>>>>>> -       scheme:  []string{"git", "https", "http", "git+ssh"},
>>>>>> +       scheme:  []string{"https", "http", "git", "git+ssh"},
>>>>>>         pingCmd: "ls-remote {scheme}://{repo}",
>>>>>>  }
>
> --
> Morten Siebuhr

[adg, 2013-10-07 23:54:31]

When does "go get" try to access VCSes on non-standard ports?

As has been discussed recently on an issue (don't have the # handy, sorry),
you can't have an import path with ":port" in it because the colon may be a
forbidden character.  Quoth the spec:

A compiler may restrict ImportPaths to non-empty strings using only
characters belonging to Unicode's L, M, N, P, and S general categories (the
Graphic characters without spaces) and may also exclude the characters
!"#$%&'()*,:;<=>?[\]^`{|} and the Unicode replacement character U+FFFD.


So is the use-case for this when we use the meta tag trick to rewrite
import paths? If so, we should only do something different when the
rewritten import path has a :port, rather than changing the behavior in all
cases.

Andrew

[rsc, 2013-10-09 01:13:25]

Nothing here is changing for Go 1.2.

[gobot, 2013-12-20 16:26:06]

Replacing golang-dev with golang-codereviews.

[gobot, 2013-12-20 16:26:06]

Replacing golang-dev with golang-codereviews.

[gobot, 2014-01-09 21:16:50]

R=adg@golang.org (assigned by bradfitz@golang.org)

[gobot, 2014-07-16 10:51:42]

R=close (assigned by dave@cheney.net)