net/http: allow customizing proxy CONNECT url #45460

liuxingbaoyu · 2021-04-08T22:49:54Z

Update May 5 2021: The current proposed API is in #45460 (comment) - rsc

What version of Go are you using (`go version`)?

go version go1.16 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

go env Output

set GO111MODULE=auto
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\流星暴雨\AppData\Local\go-build
set GOENV=C:\Users\流星暴雨\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=E:\Eternal\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=E:\Eternal\go;
set GOPRIVATE=
set GOPROXY=https://goproxy.io,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.16
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\ 流星暴雨\AppData\Local\Temp\go-build1194816388=/tmp/go-build -gno-record-gcc-switches

What did you do?

I want to use a proxy to access HTTPS websites and customize the backend IP address

What did you expect to see?

Is there some way to customize the URL of the CONNECT request

What did you see instead?

Both CONNECT request and SSL handshake use cm.targetAddr

go/src/net/http/transport.go

Lines 1667 to 1672 in 2ebe77a

    
           connectReq := &Request{ 
        
           	Method: "CONNECT", 
        
           	URL:    &url.URL{Opaque: cm.targetAddr}, 
        
           	Host:   cm.targetAddr, 
        
           	Header: hdr, 
        
           }

go/src/net/http/transport.go

Lines 1725 to 1729 in 2ebe77a

    
           if cm.proxyURL != nil && cm.targetScheme == "https" { 
        
           	if err := pconn.addTLS(ctx, cm.tlsHost(), trace); err != nil { 
        
           		return nil, err 
        
           	} 
        
           }

go/src/net/http/transport.go

Lines 1838 to 1846 in 2ebe77a

    
           // tlsHost returns the host name to match against the peer's 
        
           // TLS certificate. 
        
           func (cm *connectMethod) tlsHost() string { 
        
           	h := cm.targetAddr 
        
           	if hasPort(h) { 
        
           		h = h[:strings.LastIndex(h, ":")] 
        
           	} 
        
           	return h 
        
           }

The text was updated successfully, but these errors were encountered:

seankhliao · 2021-04-09T08:29:03Z

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

liuxingbaoyu · 2021-04-09T13:33:07Z

@seankhliao Oh no, I'm sorry that I didn't express it clearly. I read the relevant code and did not find a way to customize the CONNECT request. Perhaps this requires a new API.

seankhliao · 2021-04-09T13:48:41Z

what exactly do you want to do? change the address the of the proxy while keeping the hostname?

liuxingbaoyu · 2021-04-09T14:09:28Z

When an HTTPS request is sent through an HTTP proxy, it will first send a CONNECT request to establish a TCP connection, and then it will start a handshake with the SSL of the website. If a website has multiple resolution addresses, such as using a CDN, you can use the specified backend IP to access the website by specifying the CONNECT URL. I hope there is an API that can customize the CONNECT URL or customize the entire CONNECT request.
Similar to #13290

seankhliao · 2021-04-09T14:34:06Z

Right, different values for CONNECT xxx (for connecting) and Host: yyy (for TLS)

seankhliao · 2021-04-09T14:39:22Z

cc @bradfitz @neild @fraenkel

rsc · 2021-04-21T17:57:53Z

The code that sets up the CONNECT does:

connectReq := &Request{
	Method: "CONNECT",
	URL:    &url.URL{Opaque: cm.targetAddr},
	Host:   cm.targetAddr,
	Header: hdr,
}

If hdr["Host"] is set, then it sounds like we should use it instead of cm.targetAddr to set up connectReq.Host. That is, add:

if host, ok := hdr["Host"]; ok {
    if len(host) >= 1 {
        connectReq.Host = host[0]
    } else {
        connectReq.Host = ""
    }
}

Does anyone have any objections to this?

rsc · 2021-04-21T18:02:55Z

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

fraenkel · 2021-04-21T19:48:48Z

Sorry but I am confused on what we are trying to solve.

We have a request to some target site, www.target.org, and we are going through some Proxy. The actual request has a URL/host of www.target.org. It really doesn't matter what the host header is. The Request will ignore the Host header and generate its own based, https://github.com/golang/go/blob/master/src/net/http/request.go#L570

A proxy is required to ignore the host header if it receives an absolute uri, https://tools.ietf.org/html/rfc7230#section-5.4

When a proxy receives a request with an absolute-form of
   request-target, the proxy MUST ignore the received Host header field
   (if any) and instead replace it with the host information of the
   request-target.

rsc · 2021-04-28T17:33:52Z

@fraenkel, that's for a standard HTTP proxy, not for some custom thing serving CONNECT requests.
Do you object to providing this functionality? It seems unobjectionable to us.

fraenkel · 2021-04-28T18:44:14Z

@rsc sorry my bad.
I missed the fact that we were talking about the ProxyConnectHeaders.
You will also have to reset connectReq.URL.Opaque = "" to have it appear on the CONNECT line.

fraenkel · 2021-04-28T18:49:44Z

Probably easier to rewrite the url since the host is only used of there is no opaque bit in the url.

fraenkel · 2021-04-28T18:57:54Z

Sorry I confused myself again. Ignore what I said. I need to double check my test case which didn't see the host header on the other side.

fraenkel · 2021-04-28T19:03:24Z

I forgot that ReadRequest deletes the host header. So as long as the other side isn't using Go, this should work.

rsc · 2021-05-05T18:26:27Z

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

rsc · 2021-05-12T18:02:02Z

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

gopherbot · 2022-09-19T11:14:34Z

Change https://go.dev/cl/431756 mentions this issue: net/http: allow customizing proxy CONNECT url

neild · 2022-09-19T17:11:38Z

if host, ok := hdr["Host"]; ok {
    if len(host) >= 1 {
        connectReq.Host = host[0]
    } else {
        connectReq.Host = ""
    }
}

I don't believe this is correct.

RFC 9110 9.3.6 states that a proxy should connect to the "server identified by the request target". It doesn't say anything about the Host header that I can see, and I can't find any examples of using different values in the target and the Host header.

I did find a 2009 ietf-http-wg thread discussing the role of the Host header in CONNECT requests which seems to have concluded that the header is required, but that proxies should either ignore it or validate that it exactly matches the request target.

That said, the general idea of setting the target of the proxy request via the proxy connect headers seems reasonable; we should just set both the request target and the host header.

seankhliao closed this as completed Apr 9, 2021

seankhliao reopened this Apr 9, 2021

seankhliao changed the title ~~net/http: use custom backend IP address in https request with proxy~~ proposal: net/http: allow customizing proxy CONNECT url Apr 9, 2021

gopherbot added this to the Proposal milestone Apr 9, 2021

gopherbot added the Proposal label Apr 9, 2021

ianlancetaylor added this to Incoming in Proposals (old) Apr 14, 2021

rsc moved this from Incoming to Active in Proposals (old) Apr 21, 2021

rsc mentioned this issue Apr 21, 2021

proposal: review meeting minutes #33502

Open

rsc moved this from Active to Likely Accept in Proposals (old) May 5, 2021

rsc added the Proposal-FinalCommentPeriod label May 5, 2021

rsc moved this from Likely Accept to Accepted in Proposals (old) May 12, 2021

rsc changed the title ~~proposal: net/http: allow customizing proxy CONNECT url~~ net/http: allow customizing proxy CONNECT url May 12, 2021

rsc added the Proposal-Accepted label May 12, 2021

rsc removed this from the Proposal milestone May 12, 2021

rsc added this to the Backlog milestone May 12, 2021

rsc removed the Proposal-FinalCommentPeriod label Jul 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: allow customizing proxy CONNECT url #45460

net/http: allow customizing proxy CONNECT url #45460

liuxingbaoyu commented Apr 8, 2021 •

edited by rsc

seankhliao commented Apr 9, 2021

liuxingbaoyu commented Apr 9, 2021

seankhliao commented Apr 9, 2021

liuxingbaoyu commented Apr 9, 2021 •

edited

seankhliao commented Apr 9, 2021

seankhliao commented Apr 9, 2021

rsc commented Apr 21, 2021

rsc commented Apr 21, 2021

fraenkel commented Apr 21, 2021

rsc commented Apr 28, 2021

fraenkel commented Apr 28, 2021

fraenkel commented Apr 28, 2021

fraenkel commented Apr 28, 2021

fraenkel commented Apr 28, 2021

rsc commented May 5, 2021

rsc commented May 12, 2021

gopherbot commented Sep 19, 2022

neild commented Sep 19, 2022

net/http: allow customizing proxy CONNECT url #45460

net/http: allow customizing proxy CONNECT url #45460

Comments

liuxingbaoyu commented Apr 8, 2021 • edited by rsc

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

seankhliao commented Apr 9, 2021

liuxingbaoyu commented Apr 9, 2021

seankhliao commented Apr 9, 2021

liuxingbaoyu commented Apr 9, 2021 • edited

seankhliao commented Apr 9, 2021

seankhliao commented Apr 9, 2021

rsc commented Apr 21, 2021

rsc commented Apr 21, 2021

fraenkel commented Apr 21, 2021

rsc commented Apr 28, 2021

fraenkel commented Apr 28, 2021

fraenkel commented Apr 28, 2021

fraenkel commented Apr 28, 2021

fraenkel commented Apr 28, 2021

rsc commented May 5, 2021

rsc commented May 12, 2021

gopherbot commented Sep 19, 2022

neild commented Sep 19, 2022

liuxingbaoyu commented Apr 8, 2021 •

edited by rsc

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

liuxingbaoyu commented Apr 9, 2021 •

edited