Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: client.Do(request) panics in time.stopTimer #38219

Closed
codekhol opened this issue Apr 2, 2020 · 7 comments
Closed

net/http: client.Do(request) panics in time.stopTimer #38219

codekhol opened this issue Apr 2, 2020 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@codekhol
Copy link

codekhol commented Apr 2, 2020 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.14.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build562962017=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Unfortunately it's not possible to post the code here but it should be pretty straightforward to reproduce this.

My application makes concurrent GET requests to another API at around 280 req/sec using goroutines. This application makes about 3 million requests to the API every day which takes about 4-5 hours to finish, however when using the latest version of Go it panics after 10-120 minutes of running (the length of run before panic is not deterministic). Rolling back to Go 1.13.5 fixes this issue.

I have seen other threads here that refer to the same issue in older versions of Go, however this seems to have re-appeared in 1.14.1 again. I'm not sure if it's for a different reason this time.

What did you expect to see?

200 OK

What did you see instead?

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x451cad]

goroutine 646692 [running]:
time.stopTimer(0xc00035fa48, 0x0)
	/usr/local/go/src/runtime/time.go:224 +0x2b
time.(*Timer).Stop(...)
	/usr/local/go/src/time/sleep.go:78
net/http.(*http2ClientConn).roundTrip(0xc001e68780, 0xc001e8c900, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:7480 +0x9f
net/http.(*http2Transport).RoundTripOpt(0xc0001965a0, 0xc001e8c900, 0x9b3800, 0xc0001af080, 0xc000809770, 0x5)
	/usr/local/go/src/net/http/h2_bundle.go:6948 +0x16f
net/http.(*http2Transport).RoundTrip(...)
	/usr/local/go/src/net/http/h2_bundle.go:6909
net/http.http2noDialH2RoundTripper.RoundTrip(0xc0001965a0, 0xc001e8c900, 0xb53460, 0xc0001965a0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:9103 +0x3e
net/http.(*Transport).roundTrip(0xf4e580, 0xc001e8c900, 0xc002c91e60, 0xc001080840, 0x40c7f8)
	/usr/local/go/src/net/http/transport.go:515 +0xd94
net/http.(*Transport).RoundTrip(0xf4e580, 0xc001e8c900, 0xf4e580, 0x0, 0x0)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc001e8c900, 0xb52420, 0xf4e580, 0x0, 0x0, 0x0, 0xc00073c2f8, 0xc00002a015, 0x1, 0x0)
	/usr/local/go/src/net/http/client.go:252 +0x43e
net/http.(*Client).send(0xf58ba0, 0xc001e8c900, 0x0, 0x0, 0x0, 0xc00073c2f8, 0x0, 0x1, 0xc001e8c900)
	/usr/local/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).do(0xf58ba0, 0xc001e8c900, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:699 +0x44a
net/http.(*Client).Do(...)
	/usr/local/go/src/net/http/client.go:567
@andybons andybons changed the title http client.Do(request) panics in goroutines net/http: client.Do(request) panics in goroutines Apr 2, 2020
@andybons
Copy link
Member

andybons commented Apr 2, 2020

I have seen other threads here that refer to the same issue in older versions of Go, however this seems to have re-appeared in 1.14.1 again. I'm not sure if it's for a different reason this time.

Can you point to these threads/issues?

@andybons
Copy link
Member

andybons commented Apr 2, 2020

@bradfitz @bcmills @rsc

@andybons andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 2, 2020
@andybons andybons added this to the Unplanned milestone Apr 2, 2020
@bcmills
Copy link
Contributor

bcmills commented Apr 2, 2020

This sounds like #37400 (specifically, the symptom reported in #37490), but the fix for that (by @ianlancetaylor) was released in Go 1.14.1 (#37494).

@bcmills bcmills modified the milestones: Unplanned, Backlog Apr 2, 2020
@bcmills bcmills changed the title net/http: client.Do(request) panics in goroutines net/http: client.Do(request) panics in time.stopTimer Apr 2, 2020
@bcmills bcmills modified the milestones: Backlog, Go1.15 Apr 2, 2020
@bcmills
Copy link
Contributor

bcmills commented Apr 2, 2020

Milestoning to 1.15 based on similarlity to #37400.

@bcmills bcmills mentioned this issue Apr 3, 2020
Closed
@OneOfOne
Copy link
Contributor

OneOfOne commented Apr 3, 2020

IMO this should be 1.14.2, this is a massive regression from 1.13.x, I ran into the same issue with the exact backtrace.

@ianlancetaylor
Copy link
Contributor

I can tell from the line numbers that the stack backtrace in the original report is from 1.14, not 1.14.1, so I'm fairly certain that this was fixed by the fix for #37494.

@OneOfOne Do you still see this when using the 1.14.4 release? Can you show us the backtrace that you get? Thanks.

@ianlancetaylor ianlancetaylor added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 26, 2020
@ianlancetaylor ianlancetaylor modified the milestones: Go1.15, Backlog Jun 26, 2020
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators Jul 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
6 participants
@andybons @OneOfOne @ianlancetaylor @bcmills @codekhol @gopherbot