net/http: proxy returns 503 response but http client returns error #30560

nnathan · 2019-03-04T05:38:16Z

What version of Go are you using (`go version`)?

$ go version
go version go1.12 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build551463569=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I'm running a squid proxy server listening on 127.0.0.1:3128 which is running as squid user with an iptables rule that drops outgoing port 53 packets emitted by the squid user. This forces squid to respond to all proxied requests using a dns name with a 503 response carrying a header X-Squid-Error: ERR_DNS_FAIL 0.

I'm also writing a go program that performs a healthcheck when squid returns a response with X-Squid-Error header by making a proxied GET request to a URL using the default http client. The proxying is enabled using the http_proxy and https_proxy environment variables.

When querying a HTTP URL the http client returns the 503 response from squid with the X-Squid-Error header.

What did you expect to see?

When querying a HTTPS URL the http client should return the 503 response from squid.

What did you see instead?

When querying a HTTPS URL the http client returns an error and nil response, with the error returning "Get https://www.google.com: Service Unavailable".

The same query to https://www.google.com using curl with the https_proxy environment variable set returns the squid 503 response.

The text was updated successfully, but these errors were encountered:

davecheney · 2019-03-04T05:40:26Z

@nnathan can you please provide a sample program that someone else can use to reproduce the problem you are having. Thank you.

nnathan · 2019-03-04T05:49:34Z

Sure.

The program kind of relies on squid running that responds with a 503 and returns an X-Squid-Error header.

The squid.conf is:

#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8

acl SSL_ports port 443
acl SSL_ports port 2443
acl SSL_ports port 5222
acl SSL_ports port 8243
acl SSL_ports port 8280
acl SSL_ports port 9443
acl SSL_ports port 9445
acl SSL_ports port 9763
acl SSL_ports port 22
acl SSL_ports port 25
acl SSL_ports port 4120
acl SSL_ports port 4119
acl SSL_ports port 4122
acl Safe_ports port 4120
acl Safe_ports port 4119
acl Safe_ports port 4122
acl Safe_ports port 80
acl Safe_ports port 21
acl Safe_ports port 22
acl Safe_ports port 25
acl Safe_ports port 443
acl Safe_ports port 70
acl Safe_ports port 210
acl Safe_ports port 1025-65535
acl Safe_ports port 280
acl Safe_ports port 488
acl Safe_ports port 591
acl Safe_ports port 777
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
http_port 3128

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /var/spool/squid 100 16 256

# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid

#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320

To force DNS for squid to fail, I run: iptables -A OUTPUT -m owner --uid-owner squid -p udp --dport 53 -j DROP
Now the following go program runs a webserver on port 33128 which will serve a 200 response if it didn't detect a 503/X-Squid-Error response, otherwise it will serve a 500 response with a diagnostic.

main.go:

package main

import (
	"flag"
	"fmt"
	"log"
	"net/http"
)

var listenPort = flag.Int("listen_port", 33128, "health check webserver listen port")

var healthCheckURL = flag.String("health_check_url", "http://www.google.com", "URL to perform health check on")

func main() {
	flag.Parse()

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/plain; charset=utf-8")

		resp, err := http.Get(*healthCheckURL)

		if err != nil {
			log.Printf("error: %v", err)
			w.WriteHeader(http.StatusInternalServerError)
			w.Write([]byte(fmt.Sprintf("error: %v\n", err)))
			return
		}

		if resp.StatusCode == 503 {
			resp.Body.Close()

			squidErr := resp.Header.Get("X-Squid-Error")

			if squidErr != "" {
				log.Printf("squid failure error detected, X-Squid-Error: %s", squidErr)
				w.WriteHeader(http.StatusInternalServerError)
				w.Write([]byte(fmt.Sprintf("error detected: X-Squid-Error: %s\n", squidErr)))
				return
			}
		}

		w.WriteHeader(http.StatusOK)
		w.Write([]byte("Everying OK - did not detect any squid errors\n"))
	})

	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *listenPort), nil))
}

To run: http_proxy=http://127.0.0.1:3128 https_proxy=http://127.0.0.1:3128 go run main.go -health_check_url https://www.google.com

(change the https above to http to observe the different behaviours, whereby https will return a error on the GET and http will return a proper response)

With http://www.google.com:

# curl -v http://127.0.0.1:33128
* About to connect() to 127.0.0.1 port 33128 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 33128 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:33128
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< Date: Mon, 04 Mar 2019 05:54:11 GMT
< Content-Length: 46
< 
error detected: X-Squid-Error: ERR_DNS_FAIL 0
* Connection #0 to host 127.0.0.1 left intact

With https://www.google.com:

# curl -v http://127.0.0.1:33128
* About to connect() to 127.0.0.1 port 33128 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 33128 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:33128
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< Date: Mon, 04 Mar 2019 05:54:43 GMT
< Content-Length: 55
< 
error: Get https://www.google.com: Service Unavailable
* Connection #0 to host 127.0.0.1 left intact

And with https://www.google.com/ bypassing the healthcheck and using the proxy directly:

# https_proxy=http://127.0.0.1:3128/ http_proxy=http://127.0.0.1:3128/ curl -k -v https://www.google.com
* About to connect() to proxy 127.0.0.1 port 3128 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
* Establish HTTP proxy tunnel to www.google.com:443
> CONNECT www.google.com:443 HTTP/1.1
> Host: www.google.com:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 503 Service Unavailable
< Server: squid/3.5.20
< Mime-Version: 1.0
< Date: Mon, 04 Mar 2019 05:56:07 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3725
< X-Squid-Error: ERR_DNS_FAIL 0
< Vary: Accept-Language
< Content-Language: en
< 
* Received HTTP code 503 from proxy after CONNECT
* Connection #0 to host 127.0.0.1 left intact
curl: (56) Received HTTP code 503 from proxy after CONNECT

davecheney · 2019-03-04T05:56:04Z

Thank you for your reply. Can you please try to reduce the program. I suggest removing the http server by moving the logic from the anonymous HandleFunc into main.

nnathan · 2019-03-04T06:04:32Z

Yep sure.

So here is a condensed version without http server cruft from earlier:

package main

import (
	"log"
	"net/http"
)

func try(url string) {
	log.Printf("trying url: %s", url)
	resp, err := http.Get(url)

	if err != nil {
		log.Printf("error: %v", err)
		return
	}

	if resp.StatusCode == 503 {
		resp.Body.Close()

		squidErr := resp.Header.Get("X-Squid-Error")

		if squidErr != "" {
			log.Printf("squid failure error detected, X-Squid-Error: %s", squidErr)
			return
		}
	}

	log.Printf("Everything OK - no squid errors")
}

func main() {
	try("http://www.google.com")
	try("https://www.google.com")
}

Here is the output when running using the proxy:

$ http_proxy=http://127.0.0.1:3128 https_proxy=http://127.0.0.1:3128 go run main.go
2019/03/04 06:02:44 trying url: http://www.google.com
2019/03/04 06:03:19 squid failure error detected, X-Squid-Error: ERR_DNS_FAIL 0
2019/03/04 06:03:19 trying url: https://www.google.com
2019/03/04 06:03:19 error: Get https://www.google.com: Service Unavailable

davecheney · 2019-03-04T07:20:40Z

I think the problem is the mechanisms between http proxying, which issues a GET with the full URL and https proxying which uses the connect verb. Can you confirm the problem only occurs when you proxy https connections via squid?

…

On 4 Mar 2019, at 17:05, Naveen Nathan ***@***.***> wrote: Yep sure. So here is a condensed version with the http server: package main import ( "log" "net/http" ) func try(url string) { log.Printf("trying url: %s", url) resp, err := http.Get(url) if err != nil { log.Printf("error: %v", err) return } if resp.StatusCode == 503 { resp.Body.Close() squidErr := resp.Header.Get("X-Squid-Error") if squidErr != "" { log.Printf("squid failure error detected, X-Squid-Error: %s", squidErr) return } } log.Printf("Everything OK - no squid errors") } func main() { try("http://www.google.com") try("https://www.google.com") } Here is the output when running using the proxy: $ http_proxy=http://127.0.0.1:3128 https_proxy=http://127.0.0.1:3128 ./cc 2019/03/04 06:02:44 trying url: http://www.google.com 2019/03/04 06:03:19 squid failure error detected, X-Squid-Error: ERR_DNS_FAIL 0 2019/03/04 06:03:19 trying url: https://www.google.com 2019/03/04 06:03:19 error: Get https://www.google.com: Service Unavailable — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

nnathan · 2019-03-04T08:15:33Z

This problem occurs specifically when proxying https connections which uses the CONNECT verb.

What I've learned is that there are two responses: a response for the CONNECT, and a response for the (tunnelled) proxy request.

I guess it makes sense to treat a non-successful response to CONNECT as an error, however this means a loss of information of the underlying error (which in this case is embedded as an X-Squid-Error header). For my purposes this isn't a huge issue, checking the response from a regular HTTP request is sufficient.

julieqiu · 2019-03-12T20:08:04Z

/cc @bradfitz @rsc

timruffles · 2019-11-28T14:30:46Z

Also got surprised by this. The error originates when a non-200 is received for CONNECT - https://github.com/golang/go/blob/go1.13.4/src/net/http/transport.go#L1555

Agree with @nnathan that it makes sense this is considered a sub-HTTP error, but given that the errors end up with HTTP Statuses it's very confusing.

Feels like the appropriate change here is a more informative error - e.g CONNECT received status code xxx? Or a note in the top level Do(...) docs about this case if there's a fear some users rely on this behaviour and are reading the status code out of the error.

jmillikin-stripe · 2020-11-26T06:17:37Z

I've also hit this, with a similar use case -- a proxy that blocks access to certain domains and returns additional structured error info as an HTTP header on the failed CONNECT response.

It would be nice if either Do() returned a nil error in this case, or if there could be a way to recover the original Response from the returned error.

sheetjai · 2021-03-19T04:16:53Z

I am also observing similar issue in the environment, where squid is throwing 503 error.

Is their any workaround for this issue, which anyone found?

davecheney added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 4, 2019

davecheney removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 4, 2019

julieqiu added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 12, 2019

seankhliao added this to the Unplanned milestone Aug 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: proxy returns 503 response but http client returns error #30560

net/http: proxy returns 503 response but http client returns error #30560

nnathan commented Mar 4, 2019 •

edited

davecheney commented Mar 4, 2019

nnathan commented Mar 4, 2019 •

edited

davecheney commented Mar 4, 2019

nnathan commented Mar 4, 2019 •

edited

davecheney commented Mar 4, 2019 via email

nnathan commented Mar 4, 2019

julieqiu commented Mar 12, 2019

timruffles commented Nov 28, 2019

jmillikin-stripe commented Nov 26, 2020

sheetjai commented Mar 19, 2021

net/http: proxy returns 503 response but http client returns error #30560

net/http: proxy returns 503 response but http client returns error #30560

Comments

nnathan commented Mar 4, 2019 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

davecheney commented Mar 4, 2019

nnathan commented Mar 4, 2019 • edited

davecheney commented Mar 4, 2019

nnathan commented Mar 4, 2019 • edited

davecheney commented Mar 4, 2019 via email

nnathan commented Mar 4, 2019

julieqiu commented Mar 12, 2019

timruffles commented Nov 28, 2019

jmillikin-stripe commented Nov 26, 2020

sheetjai commented Mar 19, 2021

nnathan commented Mar 4, 2019 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

nnathan commented Mar 4, 2019 •

edited

nnathan commented Mar 4, 2019 •

edited