x/vgo: avoid github rate limits?

[flibustenet]

What version of Go are you using (go version)?

go version go1.10 linux/amd64 vgo:2018-02-20.1

What did you do?

vgo build with more than 6 dependencies

What did you see instead?

vgo: 403 response from api.github.com

GitHub applies fairly small rate limits to unauthenticated users, and
you appear to be hitting them....

solution

create a token and put it in .netrc

it can be annoying for new user.

[rsc]

Now that the tool is public, I will try to find out from GitHub if there's anything we can do about this. I don't really understand how things like OAuth2 apply to command-line binaries that ship to users and therefore can't hold their own secrets. I agree it's annoying.

[andybons]

Setting an env var with a personal access token will likely be their response: https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/

[flibustenet]

I believe it should not be mandatory to have a github account at all to use vgo.

[tianon]

For just getting a tag list, implementing git ls-remote --tags ... should be sufficient (and not subject to the GitHub API rate limits). That won't work for arbitrary commits, but those don't necessarily need to be metadata'd every time -- only on fetch when a version isn't already supplied AFAICT (since the .zip archive download works with a raw commit hash directly).

[ericlagergren]

How does homebrew do it? Their entire model is pulling from GitHub, right?

[josharian]

brew gives you instructions for getting and using a github app token when you hit rate limits.

[freman]

I've been using githubs graphql api to avoid ratelimits while checking repo freshness for my caching git proxy...

It's not universal across all hosts, but it works on github :D

[franciscocpg]

I believe it should not be mandatory to have a github account at all to use vgo.

Totally agree. Unless you have private repositories in github, there should be no reason to have a github account to use vgo.

I've been using githubs graphql api to avoid ratelimits

The limits are different. But they are still there.

[freman]

The limits are different. But they are still there.

True enough, but using graphql I can query every single repo we cache in one request for one API call.

Just putting it out there :)

[drewblas]

Many other tools seem to be able to use github-hosted files and repos for dependency fetching without hitting the rate limits. I've personally never experienced a GH rate limit error, so it's surely possible. But now I've hit one with my very first vgo command. So either vgo is doing something very inefficient or its design is such that it requires far too many calls. Hopefully it's fixable!

[terminatingcode]

Just FYI the 403 resulted in a panic during a downgrade call. I could not reproduce it a second time:

vgo get rsc.io/sampler@v1.2.0
vgo: finding rsc.io/sampler v1.2.0
vgo: finding rsc.io/quote v1.5.1
vgo: finding rsc.io/quote v1.5.0
vgo: finding rsc.io/quote v1.4.0
vgo: 403 response from api.github.com

GitHub applies fairly small rate limits to unauthenticated users, and
you appear to be hitting them. To authenticate, please visit
https://github.com/settings/tokens and click "Generate New Token" to
create a Personal Access Token. The token only needs "public_repo"
scope, but you can add "repo" if you want to access private
repositories too.

Add the token to your $HOME/.netrc:

    machine api.github.com login YOU password TOKEN

Sorry for the interruption.
vgo: rsc.io/quote v1.4.0: unexpected status (https://api.github.com/repos/rsc/quote/git/refs/tags/v1.4.0): 403 Forbidden
panic: unexpected status (https://api.github.com/repos/rsc/quote/git/refs/tags/v1.4.0): 403 Forbidden

goroutine 1 [running]:
golang.org/x/vgo/vendor/cmd/go/internal/mvs.Downgrade.func2(0xc4201e4213, 0xc, 0xc4201d6c50, 0x6)
	/Users/sarahconnor/go/src/golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:256 +0x662
golang.org/x/vgo/vendor/cmd/go/internal/mvs.Downgrade(0xc4201f6151, 0x20, 0x0, 0x0, 0x1597960, 0xc420367860, 0xc420367840, 0x1, 0x1, 0xc4200b6000, ...)
	/Users/sarahconnor/go/src/golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:281 +0x66b
golang.org/x/vgo/vendor/cmd/go/internal/vgo.runGet(0x17b3a60, 0xc42001e0e0, 0x1, 0x1)
	/Users/sarahconnor/go/src/golang.org/x/vgo/vendor/cmd/go/internal/vgo/get.go:121 +0xd0e
golang.org/x/vgo/vendor/cmd/go.Main()
	/Users/sarahconnor/go/src/golang.org/x/vgo/vendor/cmd/go/main.go:155 +0x82e
main.main()
	/Users/sarahconnor/go/src/golang.org/x/vgo/main.go:37 +0x25

[robertlestak]

There definitely needs to be a better long term solution that doesn't include requiring users to create a GitHub token just to build their code.

As @drewblas mentioned, this is the first time I've ever hit a GitHub access limit, and it was on my second vgo build. Nonetheless, I created and set an access token to continue testing with vgo.

However when I went to build the application in a Docker container, I ran into the same issue within the container.

So at this point, not only would using vgo require users to create and configure a GH access token for their local machines, but they would also have to modify all of their existing Dockerfiles to copy their local GH token into the container, build the code with vgo using the copied access token, then copy the output binary to a secondary container via a multi-stage build so as to not have their access token in the final container - that throws a real wrench in the workflow.

While doable, it's certainly not scalable.

On the positive side of the access token - vgo builds projects with private dependencies without issue, so that's nice. But I don't think that benefit outweighs the cost of requiring every person who ever wants to build a Go project to create a GitHub account.

[myitcv]

Not sure whether it's worth filing this here or in an separate issue; but one change introduced here by vgo is that the git config insteadof:

url.git@github.com:.insteadof=https://github.com/

is now effectively ignored (because explicit HTTP calls are used as opposed to delegating to the git command?)

This was previously my preferred approach to avoiding hitting rate limits. Not saying this is a preference per se, just adding a data point to this conversation.

[doxxx]

I'm not having any luck with the netrc workaround. I'm on Windows, so I've defined a HOME env var pointing to my user directory, and in that directory I have a _netrc file (so-named "because Windows" but I also tried .netrc) and it contains a single line: machine api.github.com login doxxx password <personal access token>

Any idea what I'm doing wrong?

[docwhat]

The ~/.netrc entry should be:

# You can put these on one line if you want, I just prefer them like this.
machine api.github.com
    login sekret-oauth-token
    password x-oauth-basic

My token (which I use for other things and already had in my ~/.netrc) has these permissions:

• read:gpg_key
• read:org
• read:public_key
• repo
• user:email

I suspect that no permissions are needed, since it is only reading information.

This is the page to manage your OAuth tokens.

[springeye]

GitHub applies fairly small rate limits to unauthenticated users, and
you appear to be hitting them. To authenticate, please visit
https://github.com/settings/tokens and click "Generate New Token" to
create a Personal Access Token. The token only needs "public_repo"
scope, but you can add "repo" if you want to access private
repositories too.

Add the token to your $HOME/.netrc:

    machine api.github.com login YOU password TOKEN

Sorry for the interruption.
vgo: import "github.com/henjue/note_api" ->
        import "github.com/henjue/note_api/api" ->
        import "github.com/henjue/note_api/sessions" ->
        import "github.com/kataras/iris/sessions/sessiondb/redis/service": unexpected status (https://api.github.com/repos/kataras/iris): 403 Forbidden

github.com/kataras/iris/sessions/sessiondb/redis/service is Multilevel directory

[doxxx]

@docwhat I tried that as well -- didn't work. Does the netrc method work on Windows?

[docwhat]

@doxxx I’m not sure. I think the file name is different (_netrc) and you’d have to set $HOME.

I would have to search the source to be sure.

[rsc]

The netrc is $HOME/.netrc on Windows too. That's a bug. If someone can tell me what the canonical netrc location is on Windows, we can fix that. It really should be working.

[doxxx]

Anybody else having trouble with this on Windows, I got it working. See #24606 (comment).

[NobbZ]

This hinders us to use vgo in our company, as it requires to put the access token in a semi public place in the CI service we use.

It would be really nice if vgo would be able to work without doing this API calls, but just by clonig the repository and then using git locally.

I mean, it has to work for self hosted git repositories as well, doesn't it?

[gopherbot]

Change https://golang.org/cl/107657 mentions this issue: cmd/go/internal/modfetch: update for new gitrepo backend

[gopherbot]

Change https://golang.org/cl/107656 mentions this issue: cmd/vgo/internal/modfetch/gitrepo: add general git repo access