net/http: client requests fail if TCP connection becomes stuck (due to NAT timeout or such) #30702

dop251 · 2019-03-09T13:12:49Z

What did you do?

I've come across this problem when using one of the DNS-to-HTTPS proxies that use the Cloudflare's 1.1.1.1 service. Every time the PC came out of hibernation the DNS requests kept timing out until I restarted the proxy.

To illustrate the issue I wrote a simple client:

package main

import (
	"fmt"
	"io"
	"io/ioutil"
	"net/http"
	"time"
)

func main() {
	client := &http.Client{
		Timeout: 1 * time.Second,
	}

	for {
		resp, err := client.Get("https://1.1.1.1")
		if err != nil {
			fmt.Printf("Error: %v\n", err)
		} else {
			io.Copy(ioutil.Discard, resp.Body)
			resp.Body.Close()
			fmt.Println("Success")
			time.Sleep(1 * time.Second)
		}
	}
}

I started the client and then after a few seconds I simulated a NAT timeout by creating a firewall rule that blocked outbound traffic on that connection:

# netstat -anp | grep 1.1.1.1
...
tcp        0      0 192.168.0.39:47782      1.1.1.1:443             ESTABLISHED 23236/t

# iptables -I OUTPUT -p tcp -d 1.1.1.1 --dport 443 --sport 47782 -j DROP

What did you expect to see?

If I run it with GODEBUG=http2client=0 I get one failed request after which the connection gets killed and subsequent requests cause a new one to be opened after which everything continues to work:

...
Success
Error: Get https://1.1.1.1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Success
Success
Success
...

What did you see instead?

Success
Error: Get https://1.1.1.1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Error: Get https://1.1.1.1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Error: Get https://1.1.1.1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Error: Get https://1.1.1.1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
...

System details

go version go1.12 linux/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/dop/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/dop/workspace"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
GOROOT/bin/go version: go version go1.12 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.12
uname -sr: Linux 4.4.0-142-generic
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.5 LTS
Release:	14.04
Codename:	trusty
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Ubuntu EGLIBC 2.19-0ubuntu6.14) stable release version 2.19, by Roland McGrath et al.
gdb --version: GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1

The text was updated successfully, but these errors were encountered:

terinjokes · 2019-12-06T08:59:40Z

I've been tracking down a similar looking issue in production tonight, where a server in a load balancer pool went away. Many Go HTTP/2 clients did not reconnect to the pool, continuously reporting instead:

net/http: request canceled (Client.Timeout exceeded while awaiting headers)

I can reproduce the errors described by @dop251 and that I observed locally by running the testcase and disrupting the network connection to 1.1.1.1, such as:

adding an iptables sport DROP rule similar to the given in the issue
connecting to a full tunnel VPN
changing network connection types (wifi -> ethernet, ethernet -> wifi)

I've expanded the testcase code to reduce the Dialer and Transport timeouts to below the Client timeout, expecting Go to recover from the situation, but this resulted in the same behavior.

Running with GODEBUG=http2debug=2:

2019/12/05 23:53:01 http2: Transport encoding header ":authority" = "1.1.1.1"
2019/12/05 23:53:01 http2: Transport encoding header ":method" = "GET"
2019/12/05 23:53:01 http2: Transport encoding header ":path" = "/"
2019/12/05 23:53:01 http2: Transport encoding header ":scheme" = "https"
2019/12/05 23:53:01 http2: Transport encoding header "accept-encoding" = "gzip"
2019/12/05 23:53:01 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2019/12/05 23:53:01 http2: Framer 0xc000316540: wrote HEADERS flags=END_STREAM|END_HEADERS stream=93 len=6
2019/12/05 23:53:11 http2: Framer 0xc000316540: wrote RST_STREAM stream=93 len=4 ErrCode=CANCEL
2019/12/05 23:53:11 RoundTrip failure: net/http: request canceled
Error: Get https://1.1.1.1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

The only timeout that seemed to change any behavior was ResponseHeaderTimeout, changing which error is received.

The client does eventually recover when the kernel marks the TCP connection as timed-out (that is, when tcp_retries2 expires), of which the defaults on Linux are very long. I'd expect a similar level and speed of recovery with HTTP/2 as Go provides with HTTP/1.

/cc @bradfitz

nathan-fiscaletti · 2020-09-21T22:47:26Z

Is there any update on this?

pims · 2022-12-30T02:13:34Z

Configuring ReadIdleTimeout as well as PingTimeout should be sufficient to solve this: https://go.googlesource.com/net/+/master/http2/transport.go

jdelrue · 2023-01-18T18:27:52Z

Which you do like this

test := &http2.Transport{
		ReadIdleTimeout: 20 * time.Millisecond,
	}

	client := http.Client{
		Timeout: 1 * time.Second,
	}
	client.Transport = test
	response, err := client.Get("https://api.myip.com/")

See golang/go#30702 Fixes #2355

Add "Xray REALITY with 'steal oneself'" example to README Step-by-step instructions for 'steal oneself' Bump github.com/quic-go/quic-go from 0.37.6 to 0.38.0 Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.37.6 to 0.38.0. - [Release notes](https://github.com/quic-go/quic-go/releases) - [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md) - [Commits](quic-go/quic-go@v0.37.6...v0.38.0) --- updated-dependencies: - dependency-name: github.com/quic-go/quic-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Bump github.com/quic-go/quic-go from 0.37.5 to 0.37.6 Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.37.5 to 0.37.6. - [Release notes](https://github.com/quic-go/quic-go/releases) - [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md) - [Commits](quic-go/quic-go@v0.37.5...v0.37.6) --- updated-dependencies: - dependency-name: github.com/quic-go/quic-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> fix(config): fix grpc cofnig parsing when service name only has one '/' char Bump github.com/quic-go/quic-go from 0.37.4 to 0.37.5 Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.37.4 to 0.37.5. - [Release notes](https://github.com/quic-go/quic-go/releases) - [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md) - [Commits](quic-go/quic-go@v0.37.4...v0.37.5) --- updated-dependencies: - dependency-name: github.com/quic-go/quic-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Update README.md A better docker image of xray fix(proxy): removed the udp payload length check when encryption is disabled refactor(deps): replace github.com/golang/protobuf with google.golang.org/protobuf Bump github.com/quic-go/quic-go from 0.37.3 to 0.37.4 Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.37.3 to 0.37.4. - [Release notes](https://github.com/quic-go/quic-go/releases) - [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md) - [Commits](quic-go/quic-go@v0.37.3...v0.37.4) --- updated-dependencies: - dependency-name: github.com/quic-go/quic-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Bump golang.org/x/net from 0.13.0 to 0.14.0 Bumps [golang.org/x/net](https://github.com/golang/net) from 0.13.0 to 0.14.0. - [Commits](golang/net@v0.13.0...v0.14.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Bump github.com/quic-go/quic-go from 0.37.1 to 0.37.3 Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.37.1 to 0.37.3. - [Release notes](https://github.com/quic-go/quic-go/releases) - [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md) - [Commits](quic-go/quic-go@v0.37.1...v0.37.3) --- updated-dependencies: - dependency-name: github.com/quic-go/quic-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Bump golang.org/x/crypto from 0.11.0 to 0.12.0 Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.11.0 to 0.12.0. - [Commits](golang/crypto@v0.11.0...v0.12.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Recognize .jsonc files automatically (#2398) Closes XTLS/Xray-core#2399 Add Furious to README/GUI Clients (#2405) Bump golang.org/x/net from 0.12.0 to 0.13.0 Bumps [golang.org/x/net](https://github.com/golang/net) from 0.12.0 to 0.13.0. - [Commits](golang/net@v0.12.0...v0.13.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Bump github.com/quic-go/quic-go from 0.36.2 to 0.37.1 Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.36.2 to 0.37.1. - [Release notes](https://github.com/quic-go/quic-go/releases) - [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md) - [Commits](quic-go/quic-go@v0.36.2...v0.37.1) --- updated-dependencies: - dependency-name: github.com/quic-go/quic-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Bump google.golang.org/grpc from 1.56.2 to 1.57.0 Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.56.2 to 1.57.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.56.2...v1.57.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> H2 transport: Abandon `client` if `client.Do(request)` failed See golang/go#30702 Fixes XTLS/Xray-core#2355 Add NekoRay and V2rayU to README/GUI Clients Closes XTLS/Xray-core#2354 CI: Add Linux loong64 build task Bump github.com/refraction-networking/utls from 1.3.2 to 1.3.3 Bumps [github.com/refraction-networking/utls](https://github.com/refraction-networking/utls) from 1.3.2 to 1.3.3. - [Release notes](https://github.com/refraction-networking/utls/releases) - [Commits](refraction-networking/utls@v1.3.2...v1.3.3) --- updated-dependencies: - dependency-name: github.com/refraction-networking/utls dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> fix: correct the logic of converting SocksAddr into net.Destination. fix: revert previous change on IsValid() function; add validation in getUoTConnection(); fix: Check if address is nil for IsValid() function. fix: Patch potential nil pointer deference in proxy::http::client::fillRequestHeader(). Bump github.com/quic-go/quic-go from 0.36.1 to 0.36.2 Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.36.1 to 0.36.2. - [Release notes](https://github.com/quic-go/quic-go/releases) - [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md) - [Commits](quic-go/quic-go@v0.36.1...v0.36.2) --- updated-dependencies: - dependency-name: github.com/quic-go/quic-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Bump google.golang.org/grpc from 1.56.1 to 1.56.2 Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.56.1 to 1.56.2. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.56.1...v1.56.2) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Refactor: Fragmentation XTLS/Xray-core@5f5ae37 XTLS/Xray-core@9122d0f Clean unnecessary code in Trojan Bump golang.org/x/net from 0.11.0 to 0.12.0 Bumps [golang.org/x/net](https://github.com/golang/net) from 0.11.0 to 0.12.0. - [Commits](golang/net@v0.11.0...v0.12.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Bump golang.org/x/crypto from 0.10.0 to 0.11.0 Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.10.0 to 0.11.0. - [Commits](golang/crypto@v0.10.0...v0.11.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Bump golang.org/x/sys from 0.9.0 to 0.10.0 Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.9.0 to 0.10.0. - [Commits](golang/sys@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: golang.org/x/sys dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

gofish · 2023-11-01T06:01:55Z

This can probably be closed as duplicate of or resolved in #31643

Signed-off-by: Asutorufa <16442314+Asutorufa@users.noreply.github.com>

ALTree changed the title ~~http/2 Client requests fail if TCP connection becomes stuck (due to NAT timeout or such)~~ net/http: client requests fail if TCP connection becomes stuck (due to NAT timeout or such) Mar 10, 2019

ALTree added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 10, 2019

ALTree added this to the Go1.13 milestone Mar 10, 2019

andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019

rsc modified the milestones: Go1.14, Backlog Oct 9, 2019

pims mentioned this issue Feb 18, 2020

Exposing more Envoy configuration knobs projectcontour/contour#2225

Closed

zygiss mentioned this issue Feb 21, 2020

store: Fails to sync with object store and stops responding to queries thanos-io/thanos#2098

Closed

mildwonkey mentioned this issue Oct 20, 2020

Error: Failed to query available provider packages: Client.Timeout exceeded while awaiting headers hashicorp/terraform#26532

Closed

This was referenced Jan 12, 2021

storage: Advice regarding behaviour when GCS is unavailable googleapis/google-cloud-go#3522

Closed

net/http, x/net/http2: TCP connection is not closed after http client request times out after missing RST (TCP) or GOAWAY(HTTP/2) frame #36026

Open

DarthSim mentioned this issue Apr 13, 2023

imgproxy stops downloading source images randomly imgproxy/imgproxy#1014

Closed

RPRX mentioned this issue Jul 21, 2023

reality h2模式切换网络断连，必须客户端重启服务 XTLS/Xray-core#2355

Closed

RPRX added a commit to XTLS/Xray-core that referenced this issue Jul 21, 2023

H2 transport: Abandon client if client.Do(request) failed

51769fd

See golang/go#30702 Fixes #2355

Asutorufa added a commit to yuhaiin/yuhaiin that referenced this issue Nov 1, 2023

try to fix golang/go#30702

ef38dfd

Signed-off-by: Asutorufa <16442314+Asutorufa@users.noreply.github.com>

Asutorufa added a commit to yuhaiin/yuhaiin that referenced this issue Nov 1, 2023

try to fix golang/go#30702

7fd466a

Signed-off-by: Asutorufa <16442314+Asutorufa@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: client requests fail if TCP connection becomes stuck (due to NAT timeout or such) #30702

net/http: client requests fail if TCP connection becomes stuck (due to NAT timeout or such) #30702

dop251 commented Mar 9, 2019

terinjokes commented Dec 6, 2019

nathan-fiscaletti commented Sep 21, 2020

pims commented Dec 30, 2022

jdelrue commented Jan 18, 2023

gofish commented Nov 1, 2023

net/http: client requests fail if TCP connection becomes stuck (due to NAT timeout or such) #30702

net/http: client requests fail if TCP connection becomes stuck (due to NAT timeout or such) #30702

Comments

dop251 commented Mar 9, 2019

What did you do?

What did you expect to see?

What did you see instead?

System details

terinjokes commented Dec 6, 2019

nathan-fiscaletti commented Sep 21, 2020

pims commented Dec 30, 2022

jdelrue commented Jan 18, 2023

gofish commented Nov 1, 2023