Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: add MaxBytesError #30715

Closed
sslavic opened this issue Mar 10, 2019 · 18 comments
Closed

net/http: add MaxBytesError #30715

sslavic opened this issue Mar 10, 2019 · 18 comments

Comments

@sslavic
Copy link

sslavic commented Mar 10, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN="/Users/foo/go/bin"
GOCACHE="/Users/foo/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/foo/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.12/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lf/rbfblwvx6rx3xhm68yksmqjwdv1dsf/T/go-build261266046=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Upon limiting the maximum request size (by using MaxBytesReader as in https://stackoverflow.com/a/28292505/381140) I wanted to specially handle too large request body error in a clean way with error type assertion, and return custom response body.
More people asked for the same feature: https://groups.google.com/forum/#!topic/golang-nuts/gMzVGDgPyrY

What did you expect to see?

I expected net/http to have RequestBodyTooLargeError defined and returned by MaxBytesReader so type assertion can be done instead of error message comparison.

What did you see instead?

Trivial error string implementation is returned https://golang.org/src/net/http/request.go#L1114

@ALTree ALTree added this to the Go1.13 milestone Mar 10, 2019
@ALTree ALTree added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Mar 10, 2019
@perillo
Copy link
Contributor

perillo commented Mar 10, 2019

There is a similar issue in the parsePostForm function: https://golang.org/src/net/http/request.go#L1158.

@rsc rsc modified the milestones: Go1.13, Go1.14 May 1, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@s3rj1k
Copy link

s3rj1k commented Dec 10, 2019

@rsc At a very least set those string as constants, current code looks ugly and stands against all the things that we love in golang.

const http.StatusRequestEntityTooLargeMessage = "http: request body too large"

@earthboundkid
Copy link
Contributor

@rsc, I think this issue fell off the tracker because it's from before the "Proposal" tag. Can this be tagged? If the proposal is accepted, I'm happy to file the CL.

@ianlancetaylor ianlancetaylor changed the title net/http: Define and export RequestBodyTooLargeError proposal: net/http: define and export RequestBodyTooLargeError Aug 11, 2021
@ianlancetaylor ianlancetaylor added Proposal and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Aug 11, 2021
@ianlancetaylor ianlancetaylor modified the milestones: Backlog, Proposal Aug 11, 2021
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Aug 11, 2021
@ianlancetaylor
Copy link
Contributor

CC @neild

@earthboundkid
Copy link
Contributor

FWIW, #41493 is a dupe with all positive feedback. I propose that MaxByteReader should return errors of the type ErrorMaxBytesExceeded struct { MaxSize int64 } that can be checked with errors.As.

@neild
Copy link
Contributor

neild commented Aug 12, 2021

A http.MaxBytesReader is defined as "similar to io.LimitReader", but "returns a non-EOF error for a Read beyond the limit".

Wrapping the incoming request body in a MaxBytesReader is the recommended way to set a limit on the size of an incoming request body. It interacts with other elements of HTTP request processing in some non-obvious ways: For example, (*Request).ParseForm sets a 10MiB limit on the size of parsed form data, except when the request Body is a MaxBytesReader.

This proposal, as I understand it, is to return a defined error when a MaxBytesReader hits the read limit.

This seems reasonable to me.

The net/http package defines only one exported error type, http.ProtocolError, which is deprecated. It defines a number of sentinel error values. Since the creator of a MaxBytesReader knows what maximum size they provided it, I think that providing this information in the error response is unnecessary, and this should be a sentinel error. Perhaps:

// ErrRequestBodyTooLarge is returned after a ReadCloser created by MaxBytesReader reads
// past its limit.
var ErrRequestBodyTooLarge = errors.New("http: request body too large")

A couple questions, and my proposed answers:

  • Does Close return ErrRequestBodyTooLarge if the limit is exceeded? Currently Close does not return an error in this case, but I think it should.
  • Does Read return ErrRequestBodyTooLarge, or an error wrapping this one? I don't have a strong opinion here. If this was a new package, I'd argue for wrapping, but given the extensive use of bare sentinels in net/http I think consistency argues for returning an unwrapped ErrRequestBodyTooLarge.

@earthboundkid
Copy link
Contributor

Since the creator of a MaxBytesReader knows what maximum size they provided it, I think that providing this information in the error response is unnecessary, and this should be a sentinel error.

Not necessarily. I want to wrap different handlers in different middleware, so that the processImage handler has a 25MB limit, but the processUser handler has a 1MB limit, for example. I want to be able to write generic error response middleware that can prompt the user with different messages for different overages.

@earthboundkid
Copy link
Contributor

What if instead of defining an error type, we add http.IsMaxBytesError(err error) (size int)? If the error isn’t from MaxBytesReader, it returns 0, otherwise it returns a positive number. It’s easier to use than a defined type and lets us hide the implementation details.

@rsc
Copy link
Contributor

rsc commented Oct 20, 2021

If it is going to be generated by MaxBytesReader, perhaps it should be a MaxBytesError?
@neild are you satisfied by the justification above for including the number of bytes?
Or should we just do ErrMaxBytes / ErrRequestBodyTooLarge?

@rsc
Copy link
Contributor

rsc commented Oct 20, 2021

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc moved this from Incoming to Active in Proposals (old) Oct 20, 2021
@earthboundkid
Copy link
Contributor

earthboundkid commented Oct 20, 2021

I would be happy with either of

type MaxBytesError int64

type MaxBytesError struct {
    Limit int64
}

The struct version is more extensible, but I don't foresee it needing extension, so int64 version is fine too.

A side-effect of this change is the secret requestTooLarger interface could be retired. I think it would probably stick around.

@rsc
Copy link
Contributor

rsc commented Oct 27, 2021

ping @neild re #30715 (comment)

@neild
Copy link
Contributor

neild commented Nov 3, 2021

I'm not really enthusiastic about increasing net/http's API surface with another type, but MaxBytesError seems reasonable. (I'd say that you can just wrap the io.ReadCloser returned by MaxBytesReader with your own type that adds whatever error you want--but there's a type assertion for the case when a Request.Body is a *maxBytesReader, so you can't actually do that).

type MaxBytesError struct {
  Limit int64
}

I don't think we should add http.IsMaxBytesError; error predicates are a perfectly fine pattern, but we don't have any in net/http now. A type fits the existing API better.

A struct is better than an int64. Better to always plan for extension rather than guess when you might need it.

We should maybe define that MaxBytesError{} is equivalent to any MaxBytesError, so you can write this if you don't care about the size:

if errors.Is(err, &http.MaxBytesError{}) { ... }

tl;dr, I'm reluctantly in favor of adding http.MaxBytesError.

@rsc
Copy link
Contributor

rsc commented Nov 3, 2021

We probably don't need the special definition at the end.
You can use

if errors.As(err, new(*http.MaxBytesError)) { ... }

earthboundkid added a commit to earthboundkid/go that referenced this issue Nov 4, 2021
@gopherbot
Copy link

Change https://golang.org/cl/361397 mentions this issue: net/http: add MaxBytesError

@rsc rsc moved this from Active to Likely Accept in Proposals (old) Nov 10, 2021
@rsc
Copy link
Contributor

rsc commented Nov 10, 2021

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@rsc rsc moved this from Likely Accept to Accepted in Proposals (old) Dec 1, 2021
@rsc
Copy link
Contributor

rsc commented Dec 1, 2021

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc changed the title proposal: net/http: define and export RequestBodyTooLargeError net/http: define and export RequestBodyTooLargeError Dec 1, 2021
@rsc rsc modified the milestones: Proposal, Backlog Dec 1, 2021
@earthboundkid
Copy link
Contributor

Issue title should be changed to "net/http: add MaxBytesError".

@dmitshur dmitshur changed the title net/http: define and export RequestBodyTooLargeError net/http: add MaxBytesError Dec 22, 2021
earthboundkid added a commit to earthboundkid/go that referenced this issue Mar 28, 2022
@golang golang locked and limited conversation to collaborators Apr 25, 2023
@dmitshur dmitshur modified the milestones: Backlog, Go1.19 Jun 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

10 participants