Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/net/http2: GOAWAY or reset by peer while read response body #47083

Closed
yushni opened this issue Jul 7, 2021 · 4 comments
Closed

x/net/http2: GOAWAY or reset by peer while read response body #47083

yushni opened this issue Jul 7, 2021 · 4 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@yushni
Copy link

yushni commented Jul 7, 2021

What version of Go are you using (go version)?

1.16.4

$ go version 

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

linux, amd64

go env Output
$ go env

What did you do?

Do many calls to the server, and in one of the messages i receive 200 OK status, read some part of body and
then get a GOAWAY or read: reset by peer error.

[2021-07-07 10:50:40.677] TRACE HTTPREQUEST: &{Method:GET URL:{CUT} Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[application/json] Appkey:[] Authorization:[Bearer {ELIDED}] Cli-Version:[2] Slb-Data-Partition-Id:[slb-osdu-dev-sis-internal-hq]] Body:{Reader:0xc0003baea0} GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:evq-digital-re.managed-osdu.cloud.slb-ds.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:0xc00098e480 Response:<nil> ctx:0xc0006abf20}

2021/07/07 10:50:40 http2: Transport encoding header ":method" = "GET"
2021/07/07 10:50:40 http2: Transport encoding header ":path" = "/CUT"
2021/07/07 10:50:40 http2: Transport encoding header ":scheme" = "https"
2021/07/07 10:50:40 http2: Transport encoding header "accept" = "application/json"
2021/07/07 10:50:40 http2: Transport encoding header "accept-encoding" = "gzip"
2021/07/07 10:50:40 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2021/07/07 10:50:40 http2: Framer 0xc000596620: wrote HEADERS flags=END_HEADERS stream=9 len=11
2021/07/07 10:50:40 http2: Framer 0xc000596620: wrote DATA flags=END_STREAM stream=9 len=0 data=""
2021/07/07 10:50:40 http2: Framer 0xc000596620: read WINDOW_UPDATE stream=9 len=4 incr=2147418111
2021/07/07 10:50:40 http2: Transport received WINDOW_UPDATE stream=9 len=4 incr=2147418111
2021/07/07 10:50:40 http2: Framer 0xc000596620: read GOAWAY len=8 LastStreamID=9 ErrCode=NO_ERROR Debug=""
2021/07/07 10:50:40 http2: Transport received GOAWAY len=8 LastStreamID=9 ErrCode=NO_ERROR Debug=""
2021/07/07 10:50:40 http2: Framer 0xc000596620: read HEADERS flags=END_HEADERS stream=9 len=300
2021/07/07 10:50:40 http2: decoded hpack field header field ":status" = "200"
2021/07/07 10:50:40 http2: decoded hpack field header field "date" = "Wed, 07 Jul 2021 10:50:40 GMT"
2021/07/07 10:50:40 http2: decoded hpack field header field "content-type" = "application/json"
2021/07/07 10:50:40 http2: decoded hpack field header field "access-control-allow-methods" = "OPTIONS, GET, POST, PUT, PATCH, DELETE"
2021/07/07 10:50:40 http2: decoded hpack field header field "cache-control" = "no-cache, no-store, must-revalidate"
2021/07/07 10:50:40 http2: decoded hpack field header field "content-security-policy" = "default-src 'none'"
2021/07/07 10:50:40 http2: decoded hpack field header field "expires" = "0"
2021/07/07 10:50:40 http2: decoded hpack field header field "pragma" = "no-cache"
2021/07/07 10:50:40 http2: decoded hpack field header field "strict-transport-security" = "max-age=15724800; includeSubDomains"
2021/07/07 10:50:40 http2: decoded hpack field header field "x-content-type-options" = "nosniff"
2021/07/07 10:50:40 http2: decoded hpack field header field "x-frame-options" = "deny"
2021/07/07 10:50:40 http2: decoded hpack field header field "x-xss-protection" = "1; mode=block"
2021/07/07 10:50:40 http2: Transport received HEADERS flags=END_HEADERS stream=9 len=300
2021/07/07 10:50:40 http2: Framer 0xc000596620: read DATA stream=9 len=4334 data="ath\":\"01fa09bc82s7t4qrxzyqns5seg/1/BOX_10_IX_1.S0155\"}},{\"data\":{\"type\":\"created\",\"path\":\"01fa09bc82s7t4qrxzyqns5seg/"}},{\"data\":{\"type\":\"created\",\"path\":\"01fa09bc82s7t4qrxzyqns5seg/"}},{\"data\":{\"type" (4078 bytes omitted)
2021/07/07 10:50:40 http2: Transport received DATA stream=9 len=4334 data="ath\":\"01fa09bc82s7t4qrxzyqns5seg/"}},{\"data\":{\"type\":\"created\",\"path\":\"01fa09bc82s7t4qrxzyqns5seg"}},{\"data\":{\"type\":\"created\",\"path\":\"01fa09bc82s7t4qrxzyqns5seg"}},{\"data\":{\"type" (4078 bytes omitted)

We will print response or not :)
2021/07/07 10:50:40 http2: Framer 0xc000596620: wrote WINDOW_UPDATE stream=9 len=4 incr=7680
2021/07/07 10:50:40 http2: Framer 0xc000596620: wrote WINDOW_UPDATE stream=9 len=4 incr=4846
2021/07/07 10:50:41 http2: Transport readFrame error on conn 0xc000602900: (*net.OpError) read tcp 10.132.0.28:50518->52.151.247.134:443: read: connection reset by peer
&{Status:200 OK StatusCode:200 Proto:HTTP/2.0 ProtoMajor:2 ProtoMinor:0 Header:map[Access-Control-Allow-Methods:[OPTIONS, GET, POST, PUT, PATCH, DELETE] Cache-Control:[no-cache, no-store, must-revalidate] Content-Security-Policy:[default-src 'none'] Content-Type:[application/json] Date:[Wed, 07 Jul 2021 10:50:40 GMT] Expires:[0] Pragma:[no-cache] Strict-Transport-Security:[max-age=15724800; includeSubDomains] X-Content-Type-Options:[nosniff] X-Frame-Options:[deny] X-Xss-Protection:[1; mode=block]] Body:0xc000415080 ContentLength:-1 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc000e7e800 TLS:0xc0005904d0}
failed to dump response
2021/07/07 10:50:41 http2: Framer 0xc000596620: wrote RST_STREAM stream=9 len=4 ErrCode=CANCEL
[2021-07-07 10:50:41.075] DEBUG  failed: http2: server sent GOAWAY and closed the connection; LastStreamID=9, ErrCode=NO_ERROR, debug="",  code 500
[2021-07-07 10:50:41.075] ERR0R failed .....` 

What did you expect to see?

I expected to get an error

What did you see instead?

I see status 200, start read the body and only then see error

@mknyszek mknyszek changed the title x/net/http2: GOAWAY or reset by peer while read response body. x/net/http2: GOAWAY or reset by peer while read response body Jul 7, 2021
@mknyszek mknyszek added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 7, 2021
@mknyszek mknyszek modified the milestones: Backlog, Unreleased Jul 7, 2021
@mknyszek
Copy link
Contributor

mknyszek commented Jul 7, 2021

Are you able to share a small reproducer program to help us understand the issue?

As it stands, it's hard to say what's going wrong here.

CC @neild via https://dev.golang.org/owners

@MakDon
Copy link
Contributor

MakDon commented Jul 8, 2021

Is there a possibility that you make too much request (or too frequently) so you are blocked by the server?

I've meet this problem when using http 1.1, and adding a http keep alive header solved my problem.

@yushni
Copy link
Author

yushni commented Jul 8, 2021

@MakDon, yes I do a huge amount of requests. As far as I know, http2 doesn't support the keep-alive header.
Anyway, I set up the EnableKeepAlive on the server-side and set IdleConnTimeout to 0 (unlimited) client-side.

@mknyszek it is complicated, I don't have steps on how it is possible to reproduce locally. It happens only in one working environment

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jun 12, 2022
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators Jul 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

5 participants