Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: ProxyFromEnvironment does not support socks5 #18508

Closed
bruceauyeung opened this issue Jan 4, 2017 · 24 comments
Closed

net/http: ProxyFromEnvironment does not support socks5 #18508

bruceauyeung opened this issue Jan 4, 2017 · 24 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@bruceauyeung
Copy link

bruceauyeung commented Jan 4, 2017

What version of Go are you using (go version)?

go version go1.7.4 linux/amd64

What operating system and processor architecture are you using (go env)?

actually it's Manjaro

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/bruce/go_ext_path"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build134643851=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

git config --global http.proxy 'socks5://127.0.0.1:1080'
git config --global https.proxy 'socks5://127.0.0.1:1080'
export http_proxy=socks5://127.0.0.1:1080
export https_proxy=socks5://127.0.0.1:1080
go get -u -v github.com/zmb3/gogetdoc

What did you expect to see?

Git can use socks5 proxy to clone a repo, so i expect go get can use socks5 proxy to download/update package github.com/zmb3/gogetdoc

What did you see instead?

github.com/zmb3/gogetdoc (download)
Fetching https://golang.org/x/tools/go/buildutil?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/buildutil: unrecognized import path "golang.org/x/tools/go/buildutil" (https fetch: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)
Fetching https://golang.org/x/tools/go/loader?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/loader: unrecognized import path "golang.org/x/tools/go/loader" (https fetch: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)
[bruce@bruce-manjaro ~]$ set https_proxy=socks5://127.0.0.1:1080
[bruce@bruce-manjaro ~]$ go get -u -v github.com/zmb3/gogetdoc
github.com/zmb3/gogetdoc (download)
Fetching https://golang.org/x/tools/go/buildutil?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/buildutil: unrecognized import path "golang.org/x/tools/go/buildutil" (https fetch: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)
Fetching https://golang.org/x/tools/go/loader?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/loader: unrecognized import path "golang.org/x/tools/go/loader" (https fetch: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)

@davecheney
Copy link
Contributor

davecheney commented Jan 4, 2017 via email

@josharian
Copy link
Contributor

From a quick glance, looks like maybe cmd/go needs to use http.ProxyFromEnvironment.

@josharian josharian changed the title go does not recognize socks5 proxy cmd/go: get does not use socks5 proxy Jan 4, 2017
@davecheney
Copy link
Contributor

davecheney commented Jan 4, 2017 via email

@davecheney
Copy link
Contributor

davecheney commented Jan 4, 2017 via email

@bruceauyeung
Copy link
Author

i tried to find if there is a spec for http.proxy, but found nothing.

@bradfitz
Copy link
Contributor

bradfitz commented Jan 4, 2017

Yes, we don't support SOCK5 proxies in the standard library and thus we don't support it in the cmd/go tool.

We have SOCK5 support elsewhere, so we could vendor it into std at least for use by cmd/go, but it's not clear how much it matters. Is SOCK5 still widely used?

@bruceauyeung
Copy link
Author

@davecheney useful informations that i can found about http.proxy is http://unix.stackexchange.com/a/212972

@bradfitz In China, shadowsocks is used widely

@davecheney
Copy link
Contributor

davecheney commented Jan 4, 2017 via email

@bradfitz
Copy link
Contributor

bradfitz commented Jan 4, 2017 via email

@rsc
Copy link
Contributor

rsc commented Jan 4, 2017

cmd/go does use http.ProxyFromEnvironment, because http.DefaultTransport does.

http.ProxyFromEnvironment does not support socks5. @bradfitz, where is the socks5 code we have elsewhere? It seems like it should be a tiny amount of code (under 100 lines plus tests) to proxy TCP connections into socks5. Assuming it really is small, I would not object to putting that into the standard library, on by default. It's very useful combined with ssh -D. I used to use that and tsocks, but Go doesn't play well with tsocks if I remember correctly, and tsocks is a bit of a hack.

As @davecheney says, at the least we should give a better error. But it would be nice to add support.

@rsc rsc added this to the Go1.9Early milestone Jan 4, 2017
@rsc rsc changed the title cmd/go: get does not use socks5 proxy net/http: ProxyFromEnvironment does not support socks5 Jan 4, 2017
@bradfitz
Copy link
Contributor

bradfitz commented Jan 4, 2017

@rsc, SOCK5 support is at https://godoc.org/golang.org/x/net/proxy#SOCKS5

@bradfitz bradfitz added the NeedsFix The path to resolution is known, but the work has not been done. label Jan 4, 2017
@iamzhout
Copy link

iamzhout commented Jan 5, 2017

@bruceauyeung Just suggest to use privoxy / squid alike tool to setup a local http proxy server over socks5 one to get workaround of this issue.

@bradfitz it would be useful if socks proxy could be supported. As to me it would also be ok if not as for the workaround I mentioned above.

@bruceauyeung
Copy link
Author

@bruceauyeung Just suggest to use privoxy / squid alike tool to setup a local http proxy server over socks5 one to get workaround of this issue.

@iamzhout thanks for your tip. now i go get packages through a http proxy (XX-Net), although it's slow.

@iamzhout
Copy link

iamzhout commented Jan 5, 2017

@bradfitz actually there is another issue #18519 which really bothers me a lot. Hope there would be a fix or workaround also.

@davecheney
Copy link
Contributor

davecheney commented Jan 5, 2017

@iamzhout it looks like your employer is MITM'ing your SSL communications.

Go get is just a wrapper over git, I recommend git cloning that repository directly.

@iamzhout
Copy link

iamzhout commented Jan 5, 2017

@bruceauyeung, I do suggest you to give privoxy/squid a try, as the speed should be the same as to your socks5 one. and the configuration is really simple, take privoxy as an example, you just add below line to your /etc/privoxy/config file, and then do service privoxy restart, all done.

listen-address  0.0.0.0:12345
forward-socks5  /  your-socks5-ip:port  .

@icexin
Copy link

icexin commented Jan 12, 2017

@bruceauyeung You may want this github.com/icexin/sockhttp. It's a small tool written in go that makes a HTTP proxy over SOCK5.

@bruceauyeung
Copy link
Author

@icexin i'll give it a shot, thanks for your work!

@mattn
Copy link
Member

mattn commented Jan 13, 2017

Maybe, implementaion will be like below.

package main

import (
	"context"
	"io"
	"log"
	"net"
	"net/http"
	"os"

	"golang.org/x/net/proxy"
)

type dialer struct {
	addr   string
	socks5 proxy.Dialer
}

func (d *dialer) DialContext(ctx context.Context, network, addr string) (net.Conn, error) {
	// TODO: golang.org/x/net/proxy need to add DialContext
	return d.Dial(network, addr)
}

func (d *dialer) Dial(network, addr string) (net.Conn, error) {
	var err error
	if d.socks5 == nil {
		d.socks5, err = proxy.SOCKS5("tcp", d.addr, nil, proxy.Direct)
		if err != nil {
			return nil, err
		}
	}
	return d.socks5.Dial(network, addr)
}

func Socks5Proxy(addr string) *http.Transport {
	d := &dialer{addr: addr}
	return &http.Transport{
		DialContext: d.DialContext,
		Dial:        d.Dial,
	}
}

func main() {
	http.DefaultTransport = Socks5Proxy("127.0.0.1:1080")

	resp, err := http.Get("http://www.google.com/")
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()
	io.Copy(os.Stdout, resp.Body)
}

tested with server written in go https://github.com/armon/go-socks5

@mattn
Copy link
Member

mattn commented Jan 13, 2017

So, to add common interfaces between http_proxy and socks5, I guess we should not modify ProxyFromEnvironment. it's better to add new API for set-up proxy.

@walken-google
Copy link
Contributor

I could use this feature too, and I've been looking into implementing it.

This has probably been answered before, but how do I go about importing a golang.org package (such as golang_org/x/net/proxy) so that the net/http package can use it ? I know this would require creating a src/vendor/golang_org/x/net/proxy directory but I'm not sure if it needs any tracking metadata, if the import needs to be a separate commit, etc...

@walken-google walken-google self-assigned this Jan 21, 2017
@walken-google
Copy link
Contributor

Prototype in https://go-review.googlesource.com/35488

@gopherbot
Copy link

CL https://golang.org/cl/35488 mentions this issue.

gopherbot pushed a commit that referenced this issue Mar 2, 2017
See #18508

This commit adds http Client support for socks5 proxies.

Change-Id: Ib015f3819801da13781d5acdd780149ae1f5857b
Reviewed-on: https://go-review.googlesource.com/35488
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/41031 mentions this issue: net/http: replace SOCKS client implementation

gopherbot pushed a commit that referenced this issue Apr 6, 2018
This change replaces the vendored socks client implementation with the
bundle of golang.org/x/net/internal/socks package which contains fixes
for 19354 and 11682.

golang.org/x/net/internal/socks becomes socks_bundle.go.

At git rev 61147c4. (golang.org/cl/38278)

Updates #11682.
Updates #18508.
Updates #19354.
Fixes #19688.
Updates #21333.

Change-Id: I8cf6c3f5eb87c24685a7592be015729f84fbed77
Reviewed-on: https://go-review.googlesource.com/41031
Run-TryBot: Mikio Hara <mikioh.mikioh@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@golang golang locked and limited conversation to collaborators Aug 9, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

10 participants