Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/json: Encoder.Encode newlines can cause issues with Go HTTP clients #56012

Closed
johnmaguire opened this issue Oct 3, 2022 · 3 comments

Comments

@johnmaguire
Copy link

johnmaguire commented Oct 3, 2022

What version of Go are you using (go version)?

$ go version
go version go1.19 darwin/arm64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/jmaguire/Library/Caches/go-build"
GOENV="/Users/jmaguire/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/jmaguire/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/jmaguire/go"
GOPRIVATE="github.com/DefinedNet"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.19/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.19/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.19"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/2t/rbxbv8612sq9_rdrxhtw8rzc0000gn/T/go-build1461363701=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Filing this task primarily as documentation in case someone else runs into this.

encoding/json.Encoder.Encode writes a JSON message plus a trailing newline, as described in the docs.

encoding/json.Decoder.Decode reads a JSON message from a buffer, as described.

net/http.Client.Do states that the Body must both be read to EOF and closed in order for a connection to be reused. If this does not occur the next request to Do for that server will fail with an error.

If the Body is not both read to EOF and closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.

Looking at the implementation details of Decode, it will read 512 bytes at a time. This means that if the buffer contains a single JSON message which has a byte length that is a multiple of 512, Decoder will read the JSON message and leave the trailing newline: https://go.dev/play/p/ZHrpoVGlizA

If an HTTP client uses Decode to parse a JSON message returned by a server it will occasionally fail due to the body not being read to EOF.

What did you expect to see?

No errors returned from subsequent HTTP client requests.

What did you see instead?

Intermittent EOF errors.

@mvdan
Copy link
Member

mvdan commented Oct 3, 2022

For the newlines, see #37083.

Sounds like you should make sure you decode the entire body rather than just one JSON value. For that, see #36225.

@johnmaguire
Copy link
Author

Sounds like you should make sure you decode the entire body rather than just one JSON value. For that, see #36225.

Thanks - this is the approach we've taken to solve the problem. I appreciate the linked issue, it speaks to exactly this issue. I filed this task in hopes of helping the next person who Googles for intermittent EOF errors during HTTP requests as it's the intersection of these 3 common APIs which created the problem.

@mvdan
Copy link
Member

mvdan commented Oct 3, 2022

Thanks for confirming. I'll close this as a duplicate of #36225, then.

@mvdan mvdan closed this as not planned Won't fix, can't repro, duplicate, stale Oct 3, 2022
@golang golang locked and limited conversation to collaborators Oct 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants