Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: add GOPROXY support for blob storage #32433

Closed
carnott-snap opened this issue Jun 4, 2019 · 7 comments
Closed

cmd/go: add GOPROXY support for blob storage #32433

carnott-snap opened this issue Jun 4, 2019 · 7 comments

Comments

@carnott-snap
Copy link

What did you do?

GOPROXY=gs://goproxy go get golang.org/x/xerrors

What did you see instead?

go get golang.org/x/xerrors: invalid $GOPROXY setting: malformed URL or invalid scheme (must be http, https, file)

What would you like to see changed?

Currently GOPROXY supports http, https, file protocols. It would be nice if support could be added for blob storage mechanisms like GCS, S3, etc.

@hyangah
Copy link
Contributor

hyangah commented Jun 4, 2019

Why not running a static webserver in front of blob storage (e.g. https://cloud.google.com/storage/docs/hosting-static-website) or accessing the buckets or blob storages using the default web server the service providers provide?

@carnott-snap
Copy link
Author

carnott-snap commented Jun 4, 2019

That would certainly be a stopgap, however it adds extra infrastructure and would not allow you to use default credentials for invisible authentication. Other workarounds could involve FUSE, or some GCS proxy, but it would be nice to have native support. Some of this desire comes out of the lack of GOAUTH support, see #26232, however it seems somewhat orthogonal.

This is something that other build systems have added support for, e.g. gradle. While it would add provider specific authentication logic and pull in external dependencies, this could be mitigated by shelling out to a binary, a la git helpers, and allow for more protocol flexibility.

@bcmills
Copy link
Contributor

bcmills commented Jun 6, 2019

Native gs:// support in the go command would require the go command to link in the various blob-storage clients for everybody, not just the folks who need it.

I agree with @hyangah that a blob-to-HTTP proxy seems like the right solution. It should be fairly trivial to write one that you can run locally on your machine.

If you really don't want to write your own, you could also consider a feature request with one of the existing private-proxy implementations.

CC @jayconrod @marwan-at-work.

@bcmills
Copy link
Contributor

bcmills commented Jun 6, 2019

Shelling out to a binary doesn't seem particularly simpler than having a long-running local proxy, and an ephemeral binary couldn't easily cache the connections to the blob-storage backend. It seems better to stick to the mechanism we already have (GOPROXY).

@bcmills bcmills closed this as completed Jun 6, 2019
@carnott-snap
Copy link
Author

@bcmills, considering that work on #26232 has stalled, and custom transport providers could workaround the auth issue, is there interest in pursuing a go-proxy-<transport> helper implementation? (where transport:// is provided to GOPROXY)

@bcmills
Copy link
Contributor

bcmills commented Jan 22, 2020

Is there interest? I have no idea; that's part of why #26232 has stalled. (When we fixed the .netrc bug, there seemed to be a lot less interest in alternative auth mechanisms.)

If you want to take a stab at an alternative-transport proxy, I'd be happy to review the code.

@marwan-at-work
Copy link
Contributor

@bcmills my 2 cents is that I still believe #26232 would be valuable to a number of use cases. Even if it's just a slimmed down version where you can tell the Go command to pass static headers. Happy to continue the conversation there and try to contribute that feature 😄

@golang golang locked and limited conversation to collaborators Jan 21, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants