Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: ServeContent serves wrong headers if request range is invalid #50905

Open
mitar opened this issue Jan 29, 2022 · 11 comments
Open

net/http: ServeContent serves wrong headers if request range is invalid #50905

mitar opened this issue Jan 29, 2022 · 11 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@mitar
Copy link
Contributor

mitar commented Jan 29, 2022

What version of Go are you using (go version)?

go version go1.17.6 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mitar/.cache/go-build"
GOENV="/home/mitar/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/mitar/.go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mitar/.go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.6"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build622347340=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I am serving a static blob using ServeContent, setting both Content-Length and Etag:

w.Header().Set("Content-Type", "application/json")
w.Header().Set("Content-Length", strconv.Itoa(len(jsonBody)))
w.Header().Set("Etag", etag)
http.ServeContent(w, req, "", time.Time{}, bytes.NewReader(jsonBody))

Setting Etag and Content-Length is recommended way to set those headers if one wants them.

When making an invalid range request, e.g., for a blob of 586 bytes, I do:

$ curl -X GET  "localhost:8080/blob" -D - --range 1000-

What did you expect to see?

HTTP/1.1 416 Requested Range Not Satisfiable
Content-Length: 33
Content-Range: bytes */586
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Sat, 29 Jan 2022 15:49:20 GMT

invalid range: failed to overlap

What did you see instead?

HTTP/1.1 416 Requested Range Not Satisfiable
Content-Length: 586
Content-Range: bytes */586
Content-Type: text/plain; charset=utf-8
Etag: "PxeeQ4qvnmrUjKmQLetFD2Xk34YDMF86fdlp-esVkBQ"
X-Content-Type-Options: nosniff
Date: Sat, 29 Jan 2022 15:49:20 GMT

invalid range: failed to overlap
curl: (18) transfer closed with 553 bytes remaining to read

There is:

  • Invalid Content-Length header.
  • Etag does not really hold for this content, so it should be removed.

Content-Range is in fact valid, see #15798.

@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 4, 2022
@toothrot toothrot added this to the Backlog milestone Feb 4, 2022
@gopherbot
Copy link

Change https://go.dev/cl/381956 mentions this issue: net/http: set/override Content-Length for encoded range requests

@mitar
Copy link
Contributor Author

mitar commented Apr 7, 2022

The reference was by a mistake. It is unrelated (or, partially related, but I do not think enough to be linked).

@mitar
Copy link
Contributor Author

mitar commented Nov 12, 2023

@neild: Now that #50904 got merged, I think it would be great to also fix this one. Do you agree? Or is there some reason for such behavior?

mitar added a commit to mitar/go that referenced this issue Nov 21, 2023
ServeContent API is to set some headers you want to see in the response
before calling ServeContent. But if there is an error, those headers
should be removed otherwise they might confused the client.

Fixes golang#50905.
mitar added a commit to mitar/go that referenced this issue Nov 21, 2023
ServeContent API is to set some headers you want to see in the response
before calling ServeContent. But if there is an error, those headers
should be removed otherwise they might confused the client.

Fixes golang#50905.
@gopherbot
Copy link

Change https://go.dev/cl/544019 mentions this issue: net/http: remove misleading response headers on error

@neild
Copy link
Contributor

neild commented Nov 21, 2023

@neild: Now that #50904 got merged, I think it would be great to also fix this one. Do you agree? Or is there some reason for such behavior?

I agree that ServeContent should clear out irrelevant headers in error responses.

(Commented further on https://go.dev/cl/544019.)

mitar added a commit to mitar/go that referenced this issue Nov 21, 2023
ServeContent API is to set some headers you want to see in the response
before calling ServeContent. But if there is an error, those headers
should be removed otherwise they might confused the client.

Fixes golang#50905.
mitar added a commit to mitar/go that referenced this issue Nov 21, 2023
ServeContent API is to set some headers you want to see in the response
before calling ServeContent. But if there is an error, those headers
should be removed otherwise they might confused the client.

Fixes golang#50905.
@gopherbot
Copy link

Change https://go.dev/cl/554216 mentions this issue: net/http: remove Content-Length header in http.Error

gopherbot pushed a commit that referenced this issue Feb 29, 2024
Error replies to a request with an error message and HTTP code.
Delete any preexisting Content-Length header before writing the header;
if a Content-Length is present, it's probably for content that the
caller has given up on writing.

For #50905

Change-Id: Ia3d4ca008be46fa5d41afadf29ca5cacb1c47660
Reviewed-on: https://go-review.googlesource.com/c/go/+/554216
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
mitar added a commit to mitar/go that referenced this issue Mar 4, 2024
ServeContent API is to set some headers you want to see in the response
before calling ServeContent. But if there is an error, those headers
should be removed otherwise they might confuse the client.

Removing those headers is useful in general in the case of an error,
so we remove them in http.Error.

Fixes golang#50905.
mitar added a commit to mitar/go that referenced this issue Mar 4, 2024
ServeContent API is to set some headers you want to see in the response
before calling ServeContent. But if there is an error, those headers
should be removed otherwise they might confuse the client.

Removing those headers is useful in general in the case of an error,
so we remove them in http.Error.

Fixes golang#50905.
mitar added a commit to mitar/go that referenced this issue Mar 5, 2024
ServeContent API is to set some headers you want to see in the response
before calling ServeContent. But if there is an error, those headers
should be removed otherwise they might confuse the client.

Removing those headers is useful in general in the case of an error,
so we remove them in http.Error.

Fixes golang#50905.
@gopherbot
Copy link

Change https://go.dev/cl/569815 mentions this issue: net/http: set Cache-Control header only if presents on error

gopherbot pushed a commit that referenced this issue Mar 7, 2024
CL 544019 changes http.Error to remove misleading response headers.
However, it also adds new "Cache-Control" header unconditionally, which
may breaks existing clients out there, who do not expect to see the
this header in the response like test in golang.org/x/net/http2.

To keep thing backward compatible, http.Error should only add
Cache-Control header if it has been presented.

Updates #50905

Change-Id: I989e9f999a30ec170df4fb28905f50aed0267dad
Reviewed-on: https://go-review.googlesource.com/c/go/+/569815
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
@rsc
Copy link
Contributor

rsc commented Mar 15, 2024

Reverting this change because it is breaking tests and needs more careful consideration and a godebug if we decide to move forward again.

@rsc rsc reopened this Mar 15, 2024
@gopherbot
Copy link

Change https://go.dev/cl/571975 mentions this issue: net/http: revert header changes in Error

@gopherbot
Copy link

Change https://go.dev/cl/571995 mentions this issue: net/http: remove misleading response headers on error

gopherbot pushed a commit that referenced this issue Mar 15, 2024
This reverts CL 544019 and CL 569815, because they break a variety
of tests inside Google that do not expect the Cache-Control header
to be set to no-cache.

A followup CL will add this functionality back after a proposal.

For #50905.

Change-Id: Ie377bfb72ce2c77d11bf31f9617ab6db342a408a
Reviewed-on: https://go-review.googlesource.com/c/go/+/571975
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@rsc
Copy link
Contributor

rsc commented Mar 15, 2024

Proposal discussion at #66343.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants