Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net, net/http: Deadline and Timeout Fail to Close Hung Connections. #48125

Open
JustinTimperio opened this issue Sep 1, 2021 · 3 comments
Open

net, net/http: Deadline and Timeout Fail to Close Hung Connections. #48125

JustinTimperio opened this issue Sep 1, 2021 · 3 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@JustinTimperio
Copy link

What version of Go are you using (go version)?

go 1.16 
AND 
go 1.17

Does this issue reproduce with the latest release?

Yes, this has been tested on 1.17

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/justin/.cache/go-build"
GOENV="/home/justin/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/justin/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/justin/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/justin/deadlinetest/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build833553530=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Context:

I am part of a research project called "Project-Lernaean". We are working on mapping and surveying the internet for both developers and security researchers. We have been using Colly quite successfully in our scans but recently have run into a fairly obscure issue. In a variety of cases, we run into IPs that deadlock something during a page request.

Test Cases:

Two test repositories have been created to show the bug in action. One uses Colly and the other is pure net/http and net.
https://github.com/dherbst/deadlinetest (Thanks to the work of https://github.com/dherbst)
https://github.com/Project-Lernaean/CollyTimeoutError

In both cases, Deadline and Timeout setting within http.Transport fail to terminate these connections and hang indefinitely.

Transport Settings:

transport := &http.Transport{
	IdleConnTimeout:       5 * time.Second,
	MaxIdleConns:          100,
	ExpectContinueTimeout: 10 * time.Second,
	DialContext: (&net.Dialer{
		Timeout:  1 * time.Second,
		Deadline: time.Now().Add(20 * time.Second),
	}).DialContext,
}

What did you expect to see?

I would expect that Deadline would be the absolute max time a connection can take before it is forcefully terminated.

What did you see instead?

The connection never fails and remains "open" indefinitely.
@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 2, 2021
@cherrymui cherrymui added this to the Backlog milestone Sep 2, 2021
@cherrymui
Copy link
Member

I think the Timeout and Deadline on DialContext are for dialing (creating) the connection. Once the connection is created, they are not responsible for closing it. But I'm not very familiar with how this works.

cc @neild @bradfitz

@kbolino
Copy link

kbolino commented Sep 24, 2022

I think @cherrymui is correct, but that raises the question of how to set a timeout/deadline for the request properly.

The obvious answer seems to be to add a context.Context with a deadline set to the request (e.g. via http.NewRequestWithContext or adding to an existing request with its WithContext method). However, this doesn't seem to have the desired effect in all scenarios. In particular, if the client is hung in a call to Read or Write on the underlying connection, it won't (can't?) be interrupted even if it exceeds the deadline.

What I'm describing may be a separate issue, however.

@neild
Copy link
Contributor

neild commented Sep 24, 2022

net.Dialer.Timeout and net.Dialer.Deadline are indeed for the controlling the deadline of the dial operation only.

Setting a request deadline with http.NewRequestWithContext should work in all circumstances, including when reads or writes to the underlying connection are slow. If that doesn't work, we have a bug, but that does not seem to be the case here.

@k4lizen k4lizen mentioned this issue Aug 9, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@neild @kbolino @JustinTimperio @cherrymui