Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: allow GOPRIVATE to provide source repository URI #45611

Open
ohir opened this issue Apr 17, 2021 · 12 comments
Open

proposal: cmd/go: allow GOPRIVATE to provide source repository URI #45611

ohir opened this issue Apr 17, 2021 · 12 comments

Comments

@ohir
Copy link

ohir commented Apr 17, 2021

Proposal: Allow GOPRIVATE to provide source repository URI

Last updated: 2021/04/24

Rationale:

In current state of Go modules' tooling, working with private repositories without setting up a private proxy infrastructure serving modules over https is possible but cumbersome; and still is confined to git VCS only.

For git repositories it is usually done like that:

go env -w GOPRIVATE=example.com
git config --global url."git@example.com:".insteadOf "https://example.com"

This configuration works, but it has to be global (per machine/user). To my knowledge no other VCS allows for such workaround.

Related:

#40189 : add a sub command to manage authentication

Proposal:

Allow GOPRIVATE to point to the source repository for a given module path matching pattern. Let user provide vcs type, transport, and credentials.

GOPRIVATE=pattern["["vcs+transport://[userid[:password]@]host[:port][/path]"]"][,...]

GOPRIVATE=example.com/[fossil+ssh://internal.example.com]
GOPRIVATE=example.com/[git+ssh://git@internal.example.com]
GOPRIVATE=example.com/[fossil+https://internal.example.com]
GOPRIVATE=example.com/[git+https://testing.example.com/test]

If pattern ends with slash, allow URI to be constructed using parts of matched module path represented by five percent-letter variables:

   matched.pattern/path/to/module
                           ^^^^^^ %z end element of module path
                   ^^^^           %s start element after the match
                   ^^^^^^^^^^^^^^ %p part after the match
   ^^^^^^^^^^^^^^^                %m matched by (excluding ending slash) 
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ %w whole module path

example.com[git+file:///repos/%p.git]          # get from bare repos archive
example.com[fossil+file:///repos/%s-%z.fossil]    # or a bag of user fossils
github.com/myuser/[git+file:///gopath/src/%w/] # use old GOPATH as source
example.com/[git+ssh://%s@citest.example.com:2222/%p.git] # per CI domain

Reserved characters of RFC 3986 that should be literal part of produced URI must be given in percent-encoding form. Literal characters ":", "[", "]", "", and "@" can also be provided after a backslash character.


edits:

  • 2021/04/24 add reference to #‍40189; add "reserved characters encoding" note.
@gopherbot gopherbot added this to the Proposal milestone Apr 17, 2021
@seankhliao seankhliao added the GoCommand cmd/go label Apr 17, 2021
@seankhliao seankhliao changed the title proposal: cmd/go: mod: Allow GOPRIVATE to provide source repository URI proposal: cmd/go: allow GOPRIVATE to provide source repository URI Apr 17, 2021
@seankhliao
Copy link
Member

Note it has always been possible to use your.module/path.vcs as your canonical module path, directly specifying both the module root and the VCS system to use to retrieve it

@ohir
Copy link
Author

ohir commented Apr 19, 2021

Note to the readers: (answering doubts raised at irc discussion) This proposal is just about simplifying private repository setup for solo developers and small teams. As per docs GOPRIVATE affects GONOPROXY and GONOSUMDB. Ie. modules still are "downloaded" (here copied) to the GOMODCACHE as if coming from any other source.

This proposal is entirely separate to #44347.

Also, the "keep GOPATH for private repositories" attitude will better be served by #‍44649 proposal.

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Apr 21, 2021
@ianlancetaylor
Copy link
Contributor

CC @bcmills @jayconrod

@rsc
Copy link
Contributor

rsc commented Apr 21, 2021

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@marwan-at-work
Copy link
Contributor

marwan-at-work commented Apr 24, 2021

@ohir I think #40189 and this issue are two different concerns. This issue seems to be about making GOPRIVATE be smarter when authenticating with a VCS backend. While #40189 is about having a sub command that manages authenticating with a GOPROXY.

Although I agree that authenticating with private Go Modules has always been a point of friction, the syntax here seems a bit overloaded and potentially even harder for people to configure correctly.

Go, keeping a small surface area in how it interacts with VCS systems like git is a good thing in my opinion so that it's up to the user to choose the authentication mechanism (git has many ways to do so such as .netrc, config files, and credentials stores). So Go shouldn't be concerned there.

I think for that reason, a GOPROXY is a better answer which means only one server has to configure their VCS credentials correctly and everybody else wouldn't even need to have a VCS program installed on their machine to fetch private Go Modules.

However, the current state of things means that the authentication mechanism with a GOPROXY is almost as cumbersome as the authentication mechanism with a VCS sub system, hence the reason for #40189

Thanks

@ohir
Copy link
Author

ohir commented Apr 24, 2021

@marwan-at-work
Thank you for your insights.

the syntax here seems a bit overloaded and potentially even harder for people to configure correctly.

This is pretty much (except for vcs+ part) syntax used by most CLI net tools (wget, curl, ftp, even git itself). Anyway, I got an answer I asked for, thank you :)

@rsc
Copy link
Contributor

rsc commented Apr 28, 2021

GOPRIVATE is really for encoding one specific piece of data. Trying to add more data to the encoding does not seem like it is the right approach. It seems like there's a more general problem here to solve, but I'm not sure what the exact problem statement is and how much to generalize it.

/cc @matloob @jayconrod @bcmills

@ohir
Copy link
Author

ohir commented Apr 28, 2021

@rsc

for encoding one specific piece of data [...]

This specific piece of data is enough to opt-out of public, but not enough to work in private.

what the exact problem statement is.

Setting up private repositories is cumbersome for small enterprises and for solo developers. Going private needs either a fragile and error-prone configuration dispersed among at least three env variables (GOPRIVATE GOPROXY GONOPROXY), and possibly local vcs configuration — or it needs a private goproxy server like Athens to be configured and accessible.

This proposal conceives that GOPRIVATE should be the single place where small teams and solo developers can setup all things "private" — without need to think about GOPROXY order and providing external authentication tools (as proposed in #‍40189).

Then tools can access vcs repository and add missing private modules to GOMODCACHE /cache/downloads obeying goproxy protocol.

@rsc
Copy link
Contributor

rsc commented May 19, 2021

It's clear we need an answer for the underlying problem that both this issue and #40189 are trying to solve, but neither really solves it completely. We don't have a coherent plan right now. This is something the go command team hopes to look at in the next cycle. Putting on hold until then.

@rsc rsc moved this from Active to Hold in Proposals (old) May 19, 2021
@rsc
Copy link
Contributor

rsc commented May 19, 2021

Placed on hold.
— rsc for the proposal review group

@narqo
Copy link
Contributor

narqo commented May 20, 2021

@rsc @ohir

Could the extension of go.mod, described in #39536 solve the described issue with setting up and managing private repositories per project?

@Freyert
Copy link

Freyert commented Apr 13, 2022

I've been reading a lot of these issues recently. I don't know much at all about the go internals, but why is the http protocol the first protocol tried, and why doesn't it just fallback to other protocols?

Speaking specifically about Github I would wonder why use the http protocol at all if we know Github supports the git protocol and git+ssh? Plus, Github generally pushes users to use SSH keys.

Is there a proposal to just change the default for github to git+ssh instead of https? Or at least allow users to override this setting?


Update: OK someone pointed out the reason that https is preferred. If you're not a githubber and don't have SSH set you can pull any public repository without auth. Generally that is a major win.

Which is kind of covered #26134, but this whole private repo discussion is going on across a ton of different tickets.


Relevant resources:

Github repository roots forced to HTTPS

{
pathPrefix: "github.com",
regexp: lazyregexp.New(`^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`),
vcs: "git",
repo: "https://{root}",
check: noVCSSuffix,
},

If I weren't using github I think I would be able to specify the protocol?

var vcsGit = &Cmd{
Name: "Git",
Cmd: "git",
RootNames: []string{".git"},
CreateCmd: []string{"clone -- {repo} {dir}", "-go-internal-cd {dir} submodule update --init --recursive"},
DownloadCmd: []string{"pull --ff-only", "submodule update --init --recursive"},
TagCmd: []tagCmd{
// tags/xxx matches a git tag named xxx
// origin/xxx matches a git branch named xxx on the default remote repository
{"show-ref", `(?:tags|origin)/(\S+)$`},
},
TagLookupCmd: []tagCmd{
{"show-ref tags/{tag} origin/{tag}", `((?:tags|origin)/\S+)$`},
},
TagSyncCmd: []string{"checkout {tag}", "submodule update --init --recursive"},
// both createCmd and downloadCmd update the working dir.
// No need to do more here. We used to 'checkout master'
// but that doesn't work if the default branch is not named master.
// DO NOT add 'checkout master' here.
// See golang.org/issue/9032.
TagSyncDefault: []string{"submodule update --init --recursive"},
Scheme: []string{"git", "https", "http", "git+ssh", "ssh"},
// Leave out the '--' separator in the ls-remote command: git 2.7.4 does not
// support such a separator for that command, and this use should be safe
// without it because the {scheme} value comes from the predefined list above.
// See golang.org/issue/33836.
PingCmd: "ls-remote {scheme}://{repo}",
RemoteRepo: gitRemoteRepo,
Status: gitStatus,
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Hold
Development

No branches or pull requests

9 participants