Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http: unexpected behavior of http.Server with http.MaxBytesReader and streaming response #46826

Closed
jerome-laforge opened this issue Jun 19, 2021 · 3 comments

Comments

@jerome-laforge
Copy link

jerome-laforge commented Jun 19, 2021

initially on https://groups.google.com/g/golang-nuts/c/4RDmrEYJlYE

What version of Go are you using (go version)?

$ go 1.16.5

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOENV="/home/user/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/user/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/user/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.5"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1908905363=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I have program with two variants.
The first program (https://play.golang.org/p/JRyrfOh9UZC):

  • launch the http.Server,
  • limit body input (http.MaxBytesReader),
  • read each line of jsonl input stream via json.Decode
  • send it back to client

The second program (https://play.golang.org/p/WVDK4ivA-Yj) does almost the same things except it discards the response via io.Discard.

The request bodies can be find just below (please ungzip before) and they are sent with simple curl just below:

What did you expect to see?

I expect to see correct limiter behavior on the input request body stream regardless of:

  • the size body (lower or greater than 4kB, size of http chunk?)
  • send the response back or not (the io.Discard)

What did you see instead?

With body_3kB.jsonl.gz, both variant of program works as expected. You can change the max size for limiter http.MaxBytesReader. It limits correctly according max defined size. The behaviors of limiter and http.Server are correct.

With body_4kB.jsonl.gz, the 2nd program with io.Discard works correctly.

With body_4kB.jsonl.gz, the 1st program that acctually sends response back returns unexpected error (even if body is smaller than 5kB). Furthermore, json decoder always returns an error whatever the max size of http.MaxBytesReader (100kB, 1024kB ...).
The behavior is same with io.NopCloser(io.LimitReader()) instead of http.MaxBytesReader()

@jerome-laforge jerome-laforge changed the title [http] unexpected behavior http.Server with http.MaxBytesReader and streaming response http: unexpected behavior http.Server with http.MaxBytesReader and streaming response Jun 19, 2021
@jerome-laforge jerome-laforge changed the title http: unexpected behavior http.Server with http.MaxBytesReader and streaming response http: unexpected behavior of http.Server with http.MaxBytesReader and streaming response Jun 19, 2021
@seankhliao
Copy link
Member

Duplicate of #15527

@seankhliao seankhliao marked this as a duplicate of #15527 Jun 19, 2021
@seankhliao
Copy link
Member

Note this is unrelated to MaxBytesReader, only to writing a response

@jerome-laforge
Copy link
Author

As workaround, I buffer the response until I read completely the request.
In order to limit the GC and consumption of resource, I recycle the the bytes.Buffer with sync.Pool that is in charge of create bytes.Buffer with initial capacity.

@golang golang locked and limited conversation to collaborators Jun 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants