Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: internal error: connCount underflow #61794

Closed
SABERYJS opened this issue Aug 7, 2023 · 8 comments
Closed

net/http: internal error: connCount underflow #61794

SABERYJS opened this issue Aug 7, 2023 · 8 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@SABERYJS
Copy link

SABERYJS commented Aug 7, 2023

when i send too many http request (more than one hundred thousand) by http.transport, application exit, by check log,

2023/08/06 17:14:27.188633 [ants]: worker exits from panic: net/http: internal error: connCount underflow
goroutine 414 [running]:
runtime/debug.Stack()
        /usr/lib/golang/src/runtime/debug/stack.go:24 +0x65
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
        /root/go/pkg/mod/github.com/panjf2000/ants/v2@v2.8.1/worker.go:56 +0x89
panic({0xf2e2a0, 0x1217930})
        /usr/lib/golang/src/runtime/panic.go:884 +0x212
net/http.(*Transport).decConnsPerHost(0xc000002f00, {{0xc003eb83c0, 0x34}, {0x10d58e0, 0x5}, {0xc0010cc3a8, 0x13}, 0x0})
        /usr/lib/golang/src/net/http/transport.go:1476 +0x592
net/http.(*Transport).roundTrip(0xc000002f00, 0xc005bb4e00)
        /usr/lib/golang/src/net/http/transport.go:605 +0x88b
net/http.(*Transport).RoundTrip(0xc005bb4e00?, 0x121a860?)
        /usr/lib/golang/src/net/http/roundtrip.go:17 +0x19
net/http.send(0xc005bb4b00, {0x121a860, 0xc000002f00}, {0x105c1a0?, 0x4d4601?, 0x1ba1180?})
        /usr/lib/golang/src/net/http/client.go:251 +0x5f7
net/http.(*Client).send(0xc0004988a0, 0xc005bb4b00, {0xc003f21760?, 0xc00501e630?, 0x1ba1180?})
        /usr/lib/golang/src/net/http/client.go:175 +0x9b
net/http.(*Client).do(0xc0004988a0, 0xc005bb4b00)
        /usr/lib/golang/src/net/http/client.go:715 +0x8fc
net/http.(*Client).Do(...)
        /usr/lib/golang/src/net/http/client.go:581

go version go version go1.19.10 linux/amd64

following is my http transport

func getTransport() *http.Transport {
	transport := http.DefaultTransport.(*http.Transport).Clone()
	transport.MaxIdleConns = config.SieveConfig.Transport.MaxIdleConnCount
	transport.MaxConnsPerHost = config.SieveConfig.Transport.MaxConnCount
	transport.MaxIdleConnsPerHost = config.SieveConfig.Transport.MaxIdleConnCount
	transport.DisableKeepAlives = false
	//transport.IdleConnTimeout = 90 * time.Second
	transport.TLSHandshakeTimeout = 20 * time.Second
	transport.ExpectContinueTimeout = 8 * time.Second

	header := http.Header{}
	//header.Add("Connection", "close")
	transport.ProxyConnectHeader = header

	timeout := 8 * time.Second
	transport.DialContext = (&net.Dialer{
		Timeout: timeout,
	}).DialContext
	transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
	proxyurl, err := url.Parse(ProxyServiceIpProviderImpl.GetProxy())
	if err != nil {
		panic(err)
	}
	transport.Proxy = http.ProxyURL(proxyurl)
	return transport
}

transport.MaxIdleConns set to 500
transport.MaxConnsPerHost set to 500
transport.MaxIdleConnsPerHost set to 500

@seankhliao seankhliao changed the title panic: net/http: internal error: connCount underflow net/http: internal error: connCount underflow Aug 7, 2023
@mknyszek
Copy link
Contributor

mknyszek commented Aug 7, 2023

CC @neild

@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 7, 2023
@mknyszek mknyszek added this to the Backlog milestone Aug 7, 2023
@mknyszek
Copy link
Contributor

mknyszek commented Aug 7, 2023

Does this reproduce with newer versions of Go? Note that Go 1.19 is slated to become unsupported this week.

@seankhliao seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 7, 2023
@bcmills
Copy link
Contributor

bcmills commented Aug 7, 2023

See also:

@SABERYJS, have you tried building your program with the -race flag? If so, does it report any races?

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Aug 7, 2023
@mauri870
Copy link
Member

@zzkcode posted some updates at #61474

Issue still reproducible with -race and go 1.20.6

/CC @mknyszek @bcmills

@ppanphper
Copy link

ppanphper commented Aug 23, 2023

go 1.20 also encountered this problem, after sending tens of millions of requests.

MaxConnsPerHost = 4000
MaxIdleConnsPerHost = 4000

image

@ppanphper
Copy link

image

image
image

too many call close, will there be any mistakes?

image

Will it be a fault of goroutine concurrency?

image

@kajtzu
Copy link

kajtzu commented Aug 23, 2023

We stumbled on this in an earlier version of golang and it seems to happen on 1.21.0, too, and after very many requests have been made by the client. We will see if the race detector reports something.

@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

8 participants