Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Expect: 100-continue changes behaviour if server closes request body #63152

Open
rogpeppe opened this issue Sep 21, 2023 · 4 comments
Labels
NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone

Comments

@rogpeppe
Copy link
Contributor

What version of Go are you using (go version)?

$ go version
go version devel go1.21-af8f94e3c5 Tue Jul 11 21:30:51 2023 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

N/A

What did you do?

https://go.dev/play/p/wwvMPANaBh4

What did you expect to see?

The same result regardless of whether the server closes the request body or not.

What did you see instead?

When the server closes the request body, the bytes of the request body are read inappropriately, probably because the server has sent a 100 Continue response.

Although http.Handler implementations should not close the body, it's not uncommon that to happen, for example here.

Replacing the Request.Body field (for example to replace it with a limited-size reader) has the same effect as closing it in this respect.

@rogpeppe
Copy link
Contributor Author

After a bit more investigation, it seems that the following is happening:

  • req.Body.Close ends up invoking http.body.Close via http.expectContinueReader.Close
  • http.body.Close hits the doEarlyClose branch.
  • this tries to read some of the request body, which will wait indefinitely because the client is waiting for 100 Continue before sending the request body
  • the client times out after 1s (http.Transport.ExpectContinueTimeout) and transfers the request body to the server.
  • the server reads the request body and sends the 200 response
  • the client sees the 200 response

The fact that the server is waiting for the request body when the client has explicitly said that it won't be sending it seems like it's probably a bug.

@rogpeppe
Copy link
Contributor Author

@bradfitz

@mvdan
Copy link
Member

mvdan commented Sep 22, 2023

per https://dev.golang.org/owners, we should probably ping @neild :)

@rogpeppe
Copy link
Contributor Author

One other thing to note: this behaviour does not occur when the body has an explicit content length that's more than 256KiB. In that case, the code gives up without trying to read the body which causes the client to see the response immediately: https://go.dev/play/p/ajHWF161sdD

rogpeppe added a commit to cue-labs/oci that referenced this issue Sep 22, 2023
As outlined in [this Go issue](golang/go#63152)
and [this docker registry issue](distribution/distribution#4065),
failing over from single-post to post-then-put is problematic because
we can't guarantee that the body won't be consumed.

We could implement some logic to cache the request body
until the 100-continue response arrives (I have already made a PoC
to verify this), but a simpler solution, if somewhat less efficient, is always
to use the post-then-put method, which doesn't require that failover.
rogpeppe added a commit to cue-labs/oci that referenced this issue Sep 22, 2023
As outlined in [this Go issue](golang/go#63152)
and [this docker registry issue](distribution/distribution#4065),
failing over from single-post to post-then-put is problematic because
we can't guarantee that the body won't be consumed.

We could implement some logic to cache the request body
until the 100-continue response arrives (I have already made a PoC
to verify this), but a simpler solution, if somewhat less efficient, is always
to use the post-then-put method, which doesn't require that failover.
@bcmills bcmills added this to the Backlog milestone Sep 22, 2023
@thanm thanm added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Sep 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Development

No branches or pull requests

4 participants