Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: HTTP/1 transport connection broken: readLoopPeekFailLocked: <nil> #15253

Closed
owenthereal opened this issue Apr 12, 2016 · 9 comments
Closed

net/http: HTTP/1 transport connection broken: readLoopPeekFailLocked: <nil> #15253

owenthereal opened this issue Apr 12, 2016 · 9 comments
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Unplanned

Comments

@owenthereal
Copy link

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
$ go version
go version go1.6 linux/amd64
  1. What operating system and processor architecture are you using (go env)?
$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
  1. What did you do?
    If possible, provide a recipe for reproducing the error.
    A complete runnable program is good.
    A link on play.golang.org is best.

Proxying requests using httputil.ReverseProxy with a default http.Transport. A simplified version of the code is here.

  1. What did you expect to see?

We expect everything to run fine.

  1. What did you see instead?

There were errors net/http: HTTP/1 transport connection broken: readLoopPeekFailLocked: <nil>. The error happened occasionally and seemed to only happen to backend services that responds in gzip. This is the code that emitted the error.

After some investigation, we thought it might be related to keep alive that when all the buffers line up alright but EOF was seen (like from the gzip reader) and the connection was thought to be free again and was put back in the pool. Then when the next request uses that connection it saw the end of the chunked encoding. We disabled keel alive by setting DisableKeepAlives on the transport and the errors hasn't happened for the past 24 hours. We also dug around and thought it might be related to 18072ad (but don't quote us on that).

Let us know if you need more info.

/cc @dpiddy @spofdamon
@danp
Copy link
Contributor

danp commented Apr 12, 2016

@jingweno could you please also include a sample of the "Unsolicited response received on idle HTTP channel" lines? I believe they said the data was 0\r\n\r\n.

@owenthereal
Copy link
Author

@dpiddy Good call. The error was Unsolicited response received on idle HTTP channel starting with "0\r\n\r\n"; err=<nil>. It's from here.

Also it would be nice that http.Transport allows taking a log.Logger (like httputil.ReverseProxy) so that we can use our custom logger and don't easily miss any error emitted on transport (we log errors in certain format for better query).

@bradfitz
Copy link
Contributor

Does it happen at tip?

@bradfitz bradfitz added this to the Unplanned milestone Apr 12, 2016
@owenthereal
Copy link
Author

@bradfitz We'd have to try tip on our staging. Will report result back.

@bradfitz bradfitz added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 12, 2016
@owenthereal
Copy link
Author

@bradfitz Just make sure I understand your terms, does tip mean master? Or does that mean particular branches?

@bradfitz
Copy link
Contributor

Yes, master.

@bradfitz
Copy link
Contributor

Any update?

@owenthereal
Copy link
Author

@bradfitz Unfortunately, we weren't able to reproduce the issue on our staging environment...we may have to wait for 1.7 and then turn on keep alive in prod and see if it fixes it...

Any idea on top of you mind that might have caused this?

@bradfitz
Copy link
Contributor

Pretty sure this is a dup of #15446 which has more info.

@golang golang locked and limited conversation to collaborators May 13, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
4 participants
@danp @bradfitz @owenthereal @gopherbot