cmd/go: cannot specify custom port or .git suffix to self-hosted module repo #34436

bradleygore · 2019-09-20T18:39:23Z

What version of Go are you using (`go version`)?

$ go version
go version go1.13 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN="/Users/bgore/go/bin"
GOCACHE="/Users/bgore/Library/Caches/go-build"
GOENV="/Users/bgore/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY="acme.bitbucket.com"
GONOSUMDB="acme.bitbucket.com"
GOOS="darwin"
GOPATH="/Users/bgore/go"
GOPRIVATE="acme.bitbucket.com"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.13/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.13/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/bgore/git/pos-api/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/3v/d4pfdq752wj3y14x_136dvww0000gn/T/go-build706763942=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I'm using a private bitbucket host. I've read through a lot of articles and such, and have 2 primary issues that I don't seem to be able to find a good solution for in those:

custom ports
.git suffix should be ignored on module name

My complete setup is outlined below. However, the real bitbucket URL is behind a VPN so I'm going to use a URL of acme.bitbucket.com in this example.

First, I updated my ~/.gitconfig to map the URL as needed:

[url "ssh://git@bitbucket.acme.com:1234/proj/"]
	insteadOf = https://acme.bitbucket.com/

Verified it resolves by doing a simple git ls-remote https://acme.bitbucket.com/repo.

Next, I set my GOPRIVATE to acme.bitbucket.com a l go env -w GOPRIVATE=acme.bitbucket.com.

Finally, in my project I tried to get a private module by using module.git syntax to force it to use git directly isntead the module resolution via http request and html document with a special meta tag, as that won't work here due to a custom port being in play.

$ go get -v acme.bitbucket.com/foo.git
go: finding acme.bitbucket.com/foo.git latest
go: downloading acme.bitbucket.com/foo.git v0.0.0-20190920170442-d38be5733d31
go: extracting acme.bitbucket.com/foo.git v0.0.0-20190920170442-d38be5733d31
go get: acme.bitbucket.com/foo.git@v0.0.0-20190920170442-d38be5733d31: parsing go.mod:
        module declares its path as: foo
                but was required as: acme.bitbucket.com/foo.git

So, I can understand that we'll have to just make the module named as acme.bitbucket.com/foo - that's fine. What I don't get is why the .git suffix is expected to be part of the module name. If the suffix is there as a convention to tell go get how to retrieve the module, could it not just ignore that suffix when comparing to the actual declared module?

In truth, if I could just tell it to use a specific url/port for that one dependency, then we could just use the normal way where an https request is made and an html doc comes back having this meta tag:

<meta name="go-import" content="acme.bitbucket.com:1234/blah/blah git https://acme.bitbucket.com:1234/blah/blah/blah.foo.git">

What did you expect to see?

Expect that when using .git suffix on a go get URL, that .git should not be required to be part of the module name.
Ability to tell go get to use a specific URL:port for getting the meta tag back would be fantastic.

What did you see instead?

Unable to tell go get to use a specific port for self-hosted repos on non-standard port
.git suffix seems to require same suffix on module name as shown in above output from go get -v acme.bitbucket.com/foo.git

The text was updated successfully, but these errors were encountered:

seankhliao · 2019-09-21T21:19:39Z

the .git shouldn't be necessary anywhere, it shows up in the error message because that's how you passed it go get

bradleygore · 2019-09-22T03:42:09Z

@seankhliao thx for the response. If I do not use .git at the end, then the go get mechanism attempts to use the default https method so it knows how to retrieve the module. That is, it makes a request to https://acme.bitbucket.com/foo?go-get=1 and is expecting an html document in response with a specific meta tag. Given that the VCS in question here uses a non-standard port, that mechanism will not work. This is what I was referring to and describing in item 2 in the What did you expect to see? section.

bcmills · 2019-09-23T14:22:18Z

Expect that when using .git suffix on a go get URL, that .git should not be required to be part of the module name.

go get accepts a package or module path, not a URL, and the .git suffix is part of that path. (Otherwise, how could tools like goimports or go mod tidy replicate the logic that you applied during go get?)

bcmills · 2019-09-23T14:26:34Z

Ability to tell go get to use a specific URL:port for getting the meta tag back would be fantastic.

Again, the argument to go get is an import path, not a URL, and it follows the same conventions as import statements in Go code in general.

If you want to use a VCS with a specific port, you still need to serve the go-import tag via an HTTPS server on the standard port — but note that your HTTPs server can be on an entirely different server (or even domain) from the VCS server.

bcmills · 2019-09-23T14:34:13Z

@bradleygore, please try an HTTPS server using the <meta […]> tag protocol described at https://golang.org/cmd/go/#hdr-Remote_import_paths and see whether it works for yous use-case.

(There are several open-source implementations of that protocol, such as rsc.io/go-import-redirector, and in many cases a static file server suffices.)

bradleygore · 2019-09-23T15:06:50Z

@bcmills - thx for the responses.

go get accepts a package or module path, not a URL, and the .git suffix is part of that path.

If the URL from which to pull down the package/module (if not found locally) is derived from the import path, then we have to consider both uses of the path. If the .git suffix informs it how to go about requesting the package/module, then requiring that suffix to be on the module's name in the go.mod is something that seems easy enough to nix. If we don't use that suffix, then go get uses HTTPS at that same path, and it doesn't then require the module name to have an https:// prefix...

Otherwise, how could tools like goimports or go mod tidy replicate the logic that you applied during go get

There have already been new conventions made for the module support in general, such as the replace directive. Doesn't seem like too much of a stretch to come up with a convention.

...you still need to serve the go-import tag via an HTTPS server on the standard port...

and

...please try an HTTPS server using the <meta […]> tag protocol described at https://golang.org/cmd/go/#hdr-Remote_import_paths...

This is a system I'm not in control of, so I cannot just set up an https server. Otherwise, I'd have taken the easy path already. I believe my detailed description of the issue shows I already understood how it works with the meta tag served from https and all that...

I'm simply pointing out a pain point in this situation. I was able to get the owner to move the repo in question over to github and using ssh for auth, so I'm moving forward. For whatever reason, serving self-hosted bitbucket on a standard port wasn't an option for them, and it just seemed to me that may not be a unique situation so I wrote up the issue.

bcmills · 2019-09-23T15:16:24Z

There have already been new conventions made for the module support in general, such as the replace directive. Doesn't seem like too much of a stretch to come up with a convention.

This issue is almost completely orthogonal to module support. The go command in GOPATH mode accepts import paths (not URLs) too.

And we do have a convention for exactly this use-case: this is exactly what the <meta name="go-import" […]> protocol for remote import paths is for.

bcmills · 2019-09-23T15:19:27Z

This is a system I'm not in control of, so I cannot just set up an https server.

In general we expect your import path to begin with a DNS hostname that you control, for some value of “you”. (If you are part of an organization, and someone else within that organization controls the server at the name you want to use, then you may need to talk to that someone to get things configured appropriately.)

bradleygore · 2019-09-23T15:59:48Z

you may need to talk to that someone to get things configured appropriately

As described and implied, that would be how I came to understand that exposing it on the standard port isn't an option.

Given that the import mechanism allows paths and URLs, having a convention to denote a custom port to use when it's a URL instead of assuming the standard port seems like an easy/useful thing to do. I'm having trouble thinking of many things that deal with remote addresses (ssh, databases, etc..) that do not have both a) have a default port and b) provide a means of specifying a port in cases where the default isn't used.

But hey, I'm just a person trying to communicate a pain point. If the answer is won't address that's fine 😄

bcmills · 2019-09-23T16:06:38Z

Yeah, we're not planning to do this in the go command. The alternatives require a bit of configuration, but as far as we can tell the end result is simpler overall, especially if (say) the module's hosting server changes over time.

See previously #26912 (comment).

smasher164 changed the title ~~Module Resolution in Private Repo~~ cmd/go: cannot specify custom port or .git suffix to self-hosted module repo Sep 20, 2019

smasher164 added the modules label Sep 20, 2019

bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 23, 2019

bcmills closed this as completed Sep 23, 2019

golang locked and limited conversation to collaborators Sep 22, 2020

gopherbot added the FrozenDueToAge label Sep 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: cannot specify custom port or .git suffix to self-hosted module repo #34436

cmd/go: cannot specify custom port or .git suffix to self-hosted module repo #34436

bradleygore commented Sep 20, 2019

seankhliao commented Sep 21, 2019

bradleygore commented Sep 22, 2019 •

edited

bcmills commented Sep 23, 2019

bcmills commented Sep 23, 2019

bcmills commented Sep 23, 2019

bradleygore commented Sep 23, 2019

bcmills commented Sep 23, 2019 •

edited

bcmills commented Sep 23, 2019

bradleygore commented Sep 23, 2019

bcmills commented Sep 23, 2019

cmd/go: cannot specify custom port or .git suffix to self-hosted module repo #34436

cmd/go: cannot specify custom port or .git suffix to self-hosted module repo #34436

Comments

bradleygore commented Sep 20, 2019

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

seankhliao commented Sep 21, 2019

bradleygore commented Sep 22, 2019 • edited

bcmills commented Sep 23, 2019

bcmills commented Sep 23, 2019

bcmills commented Sep 23, 2019

bradleygore commented Sep 23, 2019

bcmills commented Sep 23, 2019 • edited

bcmills commented Sep 23, 2019

bradleygore commented Sep 23, 2019

bcmills commented Sep 23, 2019

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

bradleygore commented Sep 22, 2019 •

edited

bcmills commented Sep 23, 2019 •

edited