Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proposal: net/http: Customize limit on number of 1xx responses #65035

Open
Acconut opened this issue Jan 9, 2024 · 5 comments
Open

proposal: net/http: Customize limit on number of 1xx responses #65035

Acconut opened this issue Jan 9, 2024 · 5 comments
Labels
Proposal
Milestone
Proposal

Comments

@Acconut
Copy link

Acconut commented Jan 9, 2024 •
edited

Proposal Details

In HTTP, a server can send multiple responses for one requests: zero or more informational responses (1xx) and one final response (2xx, 3xx, 4xx, 5xx). Go's HTTP client is capable of receiving those informational responses and exposes them to users via net/http/httptrace.ClientTrace.Got1xxResponse. However, the client has a default limit of reading only up to 5 responses. Any additional 1xx response will trigger a net/http: too many 1xx informational responses error.

go/src/net/http/transport.go

Lines 2328 to 2329 in 8db1310

num1xx := 0 // number of informational 1xx headers received
const max1xxResponses = 5 // arbitrary bound on number of informational responses

go/src/net/http/transport.go

Lines 2354 to 2357 in 8db1310

num1xx++
if num1xx > max1xxResponses {
return nil, errors.New("net/http: too many 1xx informational responses")
}

The code comments and the original commit (d88b137) mention that the limit of 5 responses is arbitrary. If the limit is reached, the entire request is stopped and the client cannot receive the final response (2xx etc) anymore. This is problematic for applications, where the server repeatedly sends informational responses. 5 is a sensible default value for nearly all applications, but it would be helpful if this limit could be customized to allow more or even an unlimited amount of responses.

One option for implementing this, would be to add another field to the net/http.Client struct. Setting it to a zero value keeps the current limit of 5 responses, while any other non-zero value sets the limit accordingly.

Background

In the HTTP working group of the IETF we are discussing a draft on resumable uploads. We are considering including a feature where the server can repeatedly send 1xx responses to inform the client about the upload progress. In these scenarios, the client sends data in the request body and repeatedly receives progress information in the 1xx responses. This progress information can be used to release data that is buffered on the client-side.

Example

Below you can find a brief program reproducing this behavior. The client sends a request to a server which responds with 10 1xx responses:

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"net/http/httptest"
	"net/http/httptrace"
	"net/textproto"
	"time"
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		for i := 0; i < 10; i++ {
			w.Header().Set("X-Progress", fmt.Sprintf("%d%%", i*10))
			w.WriteHeader(150)
			<-time.After(100 * time.Millisecond)
		}

		w.WriteHeader(200)
	}))
	defer ts.Close()

	ctx := httptrace.WithClientTrace(context.Background(), &httptrace.ClientTrace{
		Got1xxResponse: func(code int, header textproto.MIMEHeader) error {
			fmt.Println("Progress:", header.Get("X-Progress"))

			return nil
		},
	})

	req, err := http.NewRequestWithContext(ctx, "GET", ts.URL, nil)
	if err != nil {
		log.Fatal(err)
	}

	client := ts.Client()
	res, err := client.Do(req)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(res.Status)
}

The client receives the first 5 1xx responses, but then errors out. The final response is not received by the client.

$ go run clients/go/test/example.go 
Progress: 0%
Progress: 10%
Progress: 20%
Progress: 30%
Progress: 40%
2024/01/09 10:05:56 Get "http://127.0.0.1:52073": net/http: HTTP/1.x transport connection broken: net/http: too many 1xx informational responses
exit status 1

If the limit could be raised, the client could receive all informational and the final response without an error.
@gopherbot gopherbot added this to the Proposal milestone Jan 9, 2024
@Acconut Acconut mentioned this issue Jan 9, 2024
Merged
@seankhliao
Copy link
Member

cc @neild

@royfielding
Copy link

This is a bug in the Go client. HTTP is expected to have many interim responses, depending on the nature of the request. There is no reasonable limit on number of valid interim responses. Setting a time limit overall might make sense for capacity reasons, but this default limit is nuts.

@joakime
Copy link

joakime commented Jan 16, 2024

Wow! @royfielding commenting on an HTTP user-agent bug on github. Hard to get more authoritative about the HTTP spec.

@neild
Copy link
Contributor

neild commented Jan 16, 2024

A configurable limit is too fiddly; we shouldn't require users to twiddle a knob to get reasonable behavior.

We generally consider hostile servers to be less of a concern than hostile clients, but no limit might be a bit much; a server shouldn't be able to send an arbitrary amount of data in response to a request without the client's knowledge.

Perhaps a good compromise might be something along the lines of: No more than N 1xx responses, resetting the counter after a time interval and after every response byte read. (So interleaving an arbitrary number of 1xx responses with response bytes is acceptable, as is sending 1xx responses with no data but below some rate.)

Whatever we do will need to be synchronized between the HTTP/1 and HTTP/2 paths, which both implement the same no-more-than-5 logic at this time.

@Acconut
Copy link
Author

Acconut commented Jan 23, 2024

Thanks for the comments!

we shouldn't require users to twiddle a knob to get reasonable behavior.

This would be an ideal scenario, but I am not not sure if it's possible in this case.

No more than N 1xx responses, resetting the counter after a time interval

That would be an option. For example, with this approach we could allow 5 responses in 5 seconds?

after every response byte read. (So interleaving an arbitrary number of 1xx responses with response bytes is acceptable, as is sending 1xx responses with no data but below some rate.)

I am not sure this is possible. As far as I know interim responses and the final response cannot be interleaved. When the client starts reading the final response body, no more interim responses will be sent anyway. This should be the case for HTTP/1.1 and also HTTP/2 from reading https://httpwg.org/specs/rfc9113.html#HttpFraming.

a server shouldn't be able to send an arbitrary amount of data in response to a request without the client's knowledge.

Would it be possible to reuse net/http.Transport.MaxResponseHeaderBytes for this purpose? It could be the limit on the size of all response headers combined (i.e. interim response headers plus the final response header). The default value would allow some interim responses to be sent while protecting against too many. While in use cases like mine, we could increase the limit to allow more interim responses.

Right now MaxResponseHeaderBytes is applied on a per-response basis and is reset for every interim or final response:

go/src/net/http/transport.go

Line 2358 in 704401f

pc.readLimit = pc.maxHeaderResponseSize() // reset the limit

Whatever we do will need to be synchronized between the HTTP/1 and HTTP/2 paths, which both implement the same no-more-than-5 logic at this time.

Yss, I agree on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Proposal
Projects
Proposals
Status: Incoming
Milestone
Proposal
Development

No branches or pull requests
6 participants
@neild @joakime @Acconut @royfielding @gopherbot @seankhliao