Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: transport hang #38584

Closed
kstenerud opened this issue Apr 22, 2020 · 5 comments
Closed

net/http: transport hang #38584

kstenerud opened this issue Apr 22, 2020 · 5 comments
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@kstenerud
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.13.9 linux/amd64

Does this issue reproduce with the latest release?

I can't reproduce it

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/karl/.cache/go-build"
GOENV="/home/karl/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/karl/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/karl/work/Devel/testgorm/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build561612903=/tmp/go-build -gno-record-gcc-switches"

What did you do?

It was just a standard HTTP post:

	request, err := http.NewRequest("POST", c.signingEndpoint.String(), nil)
	if err != nil {
		return "", err
	}
	request.Header.Add("Authorization", c.signingAuth)
	client := http.DefaultClient
	response, err = client.Do(request)

What did you expect to see?

I expected it to succeed or fail

What did you see instead?

It hung:

goroutine 1 [select]:
net/http.(*Transport).getConn(0xd12d60, 0xc000198300, 0x0, 0xc0003986c0, 0x5, 0xc000362a60, 0x10, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/net/http/transport.go:1239 +0x553
net/http.(*Transport).roundTrip(0xd12d60, 0xc0000fe000, 0xc0001c4900, 0xc0000f32f0, 0x700000000414378)
	/usr/local/go/src/net/http/transport.go:522 +0x603
net/http.(*Transport).RoundTrip(0xd12d60, 0xc0000fe000, 0xd12d60, 0x0, 0x0)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc0000fe000, 0xa446a0, 0xd12d60, 0x0, 0x0, 0x0, 0xc000010028, 0xd, 0x1, 0x0)
	/usr/local/go/src/net/http/client.go:250 +0x443
net/http.(*Client).send(0xd52fe0, 0xc0000fe000, 0x0, 0x0, 0x0, 0xc000010028, 0x0, 0x1, 0x7ff365c16490)
	/usr/local/go/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xd52fe0, 0xc0000fe000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:641 +0x3ce
net/http.(*Client).Do(...)
	/usr/local/go/src/net/http/client.go:509
github.com/agentclient/agent-client.(*agentClient).sendSigningRequest(0xc0000b2aa0, 0xc000318060, 0x0, 0x414378, 0x80)
	/home/karl/agent-client/agent_client.go:253 +0x24c
github.com/agentclient/agent-client.(*agentClient).CurrentAgentQuery(0xc0000b2aa0, 0x59e4c2, 0xc0001ce000, 0xa4cce0, 0xc0000aa078)
	/home/karl/agent-client/agent_client.go:69 +0xb6
github.com/agentclient/agent-client.(*agentClient).sendProfilePrologue(0xc0000b2aa0, 0xc0000f3c78, 0x4, 0x9a0201)
	/home/karl/agent-client/agent_client.go:141 +0x43
github.com/agentclient/agent-client.(*agentClient).SendProfile(0xc0000b2aa0, 0xc0003fc000, 0x9e06, 0xf000, 0x0, 0x0)
	/home/karl/agent-client/agent_client.go:223 +0x158
github.com/agentclient/agent-client.(*probe).endProfile(0xc0000fc600, 0x0, 0x0)
	/home/karl/agent-client/probe.go:455 +0x356
github.com/agentclient/agent-client.(*probe).End(0xc0000fc600, 0x0, 0x0)
	/home/karl/agent-client/probe.go:234 +0x402
github.com/agentclient/agent-client.End()
	/home/karl/agent-client/api.go:59 +0x2d
main.main()
	/home/karl/work/Devel/testapp/main.go:60 +0x2ec

Looking at the source, it seems to have hung while waiting for the transport queue to allow it to dial. This is a very simple, single goroutine program that was making a standard HTTP POST request.

It only happened once. I'd never encountered it before, and was unable to reproduce again, so it will be hard to track down what happened, but I suspect I hit an edge case in the locking and queue logic.

@andybons andybons changed the title net/http transport hang net/http: transport hang Apr 22, 2020
@andybons andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 22, 2020
@andybons andybons added this to the Unplanned milestone Apr 22, 2020
@andybons
Copy link
Member

@bradfitz @neild

@davecheney
Copy link
Contributor

@kstenerud would you be able to upgrade to Go 1.16 and confirm if the problem remains. Thank you.

@davecheney davecheney added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Feb 24, 2021
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@secmask
Copy link

secmask commented Dec 24, 2021

@kstenerud would you be able to upgrade to Go 1.16 and confirm if the problem remains. Thank you.

I got same issue, and I can't reproduce it,
go version 1.16.12

92 @ 0x43cd45 0x44df57 0x71f1e9 0x71a88b 0x701295 0x96f5ed 0x9702d2 0x6be134 0x6bdb1f 0x6bfb9f 0x96e025 0x96e000 0x96e53e 0x96b655 0x96d745 0x970791 0x474101
#	0x71f1e8	net/http.(*Transport).getConn+0x588									/opt/hostedtoolcache/go/1.16.12/x64/src/net/http/transport.go:1368
#	0x71a88a	net/http.(*Transport).roundTrip+0x7ea									/opt/hostedtoolcache/go/1.16.12/x64/src/net/http/transport.go:579
#	0x701294	net/http.(*Transport).RoundTrip+0x34									/opt/hostedtoolcache/go/1.16.12/x64/src/net/http/roundtrip.go:17

@gland1
Copy link

gland1 commented Sep 21, 2022

It happens also with golang 17.
For me it happens after mutliple connection timeout errors.
eventually it hanged forever:

goroutine 27 [select, 1469 minutes]:
net/http.(*Transport).getConn(0xc0004443c0, 0xc00042c540, {{}, 0x0, {0xc00003e0f0, 0x5}, {0xc00061e060, 0x15}, 0x0})
/usr/local/go/src/net/http/transport.go:1372 +0x5d2
net/http.(*Transport).roundTrip(0xc0004443c0, 0xc0002cc000)
/usr/local/go/src/net/http/transport.go:581 +0x774
net/http.(*Transport).RoundTrip(0xc0003b4540, 0xc0002cc000)
/usr/local/go/src/net/http/roundtrip.go:18 +0x19
golang.org/x/oauth2.(*Transport).RoundTrip(0xc0006266c0, 0xc000446500)
/root/workspace/KumoScale-CSI-3.22/pkg/mod/golang.org/x/oauth2@v0.0.0-20211104180415-d3ed0bb246c8/transport.go:55 +0x14a
net/http.send(0xc000446500, {0x15199c0, 0xc0006266c0}, {0x13304a0, 0x1, 0x0})
/usr/local/go/src/net/http/client.go:252 +0x5d8
net/http.(*Client).send(0xc00050f680, 0xc000446500, {0xc000446500, 0x2f, 0x0})
/usr/local/go/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc00050f680, 0xc000446500)
/usr/local/go/src/net/http/client.go:725 +0x908
net/http.(*Client).Do(...)
/usr/local/go/src/net/http/client.go:593

@golang golang locked and limited conversation to collaborators Sep 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

6 participants