Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: RoundTrip unexpectedly changes Request #39533

Closed
neild opened this issue Jun 11, 2020 · 7 comments
Closed

net/http: RoundTrip unexpectedly changes Request #39533

neild opened this issue Jun 11, 2020 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Milestone
Go1.15

Comments

@neild
Copy link
Contributor

neild commented Jun 11, 2020

https://go-review.googlesource.com/c/go/+/234894 causes (*http.Transport).RoundTrip to return a different *http.Request in the *http.Response than was passed to it, under some circumstances.

func main() {
        req, err := http.NewRequest("GET", "http://google.com/", bytes.NewBuffer([]byte{0, 1, 2}))
        if err != nil {
                log.Fatal(err)
        }
        resp, err := http.DefaultTransport.RoundTrip(req)
        if err != nil {
                log.Fatal(err)
        }
        fmt.Println(req == resp.Request) // false, where it used to be true.
}

The documentation for (http.Response).Request says, "Request is the request that was sent to obtain this Response." While this doesn't precisely specify that it's the exact *Request passed to RoundTrip, that's probably close enough to consider this an incorrect API change.

Discovered because this breaks code which keeps a map of *http.Requests.
@neild neild self-assigned this Jun 11, 2020
@ianlancetaylor ianlancetaylor added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Jun 11, 2020
@ianlancetaylor ianlancetaylor added this to the Go1.15 milestone Jun 11, 2020
@toothrot toothrot mentioned this issue Jun 11, 2020
Closed
@neild
Copy link
Contributor Author

neild commented Jun 11, 2020

Even prior to this change, RoundTrip would change the Request, but only when calling GetBody to rewind the body:
https://go.googlesource.com/go/+/0e617d3d5c7e89b1ad1b0285fc77314b8d056211/src/net/http/transport.go#588

Tracking the history of this behavior, I believe it originates with https://golang.org/cl/33971, which causes the HTTP2 transport to replace the request when retrying.

My first thought was that it seems odd to replace the entire *Request rather than just updating its Body field, but the http.RoundTripper documentation states:

RoundTrip should not modify the request, except for consuming and closing the Request's Body.

Replacing the *Request in RoundTrip only under unusual circumstances (when calling GetBody to reset the body) is error-prone; users may depend on the *Request passing through unchanged, and only encounter infrequent errors. I think the *Request should be replaced consistently, or never.

My inclination is to say that "never" is the right answer, since it causes the least disruption to existing users.

If that's the case, we could:

  • Change the RoundTripper contract to permit changing the request Body when GetBody is called. I think this is reasonable behavior, but I'm not certain if this would be consistent with our compatibility guarantees.
  • Temporarily replace the request Body, but put the original one back before returning.
  • Create a new Request when retrying, but return the original in the Response.

@rsc
Copy link
Contributor

rsc commented Jun 11, 2020 •
edited

Requests are read-only data, immutable. I believe it is allowed today to call RoundTrip from multiple goroutines using the same request (provided Body = NoBody or nil). Mutating the Request would break that, introducing races.

The main time that Requests change and the reason Response records the final Request is when http.Client.Do follows redirects. That's the time when you most often need to look at the Response - to find out what the final fetched URL even was (if you care). Since they can change in Do, I'm not convinced it's important they can't change in RoundTrip.

Maybe we can say that the Response returned by RoundTrip always returns the original Request? I'm not sure. It's either that or decide the tests are broken. You can't mutate in place.

/cc @bradfitz

@gopherbot
Copy link

Change https://golang.org/cl/237560 mentions this issue: net/http: make Transport.RoundTrip preserve Requests

@neild
Copy link
Contributor Author

neild commented Jun 11, 2020

We'd only need to mutate the Request if the Body is neither NoBody nor nil, which are the cases where you aren't allowed to call RoundTrip from multiple goroutines. If immutable requests are an invariant, though, then it is what it is.

@andybons andybons mentioned this issue Jul 15, 2020
Closed
@dmitshur
Copy link
Contributor

@gopherbot Please backport to Go 1.14.

This is a fixup to a previous fix (CL 242117) for an approved backport issue #39279. We can't submit that CL without this one. This needs to be backported to Go 1.14 only, since the fix is already included in Go 1.1​5.

@gopherbot
Copy link

Backport issue(s) opened: #40973 (for 1.14).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@gopherbot gopherbot mentioned this issue Aug 22, 2020
Closed
@gopherbot
Copy link

Change https://golang.org/cl/249880 mentions this issue: [release-branch.go1.14] net/http: make Transport.RoundTrip preserve Requests

gopherbot pushed a commit that referenced this issue Aug 27, 2020
@neild @dmitshur
[release-branch.go1.14] net/http: make Transport.RoundTrip preserve R…
fae8e09 
…equests

Ensure that the exact Request passed to Transport.RoundTrip
is returned in the Response. Do not replace the Request with
a copy when resetting the request body.

Updates #39533.
Fixes #40973.

Change-Id: Ie6fb080c24b0f6625b0761b7aa542af3d2411817
Reviewed-on: https://go-review.googlesource.com/c/go/+/237560
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-on: https://go-review.googlesource.com/c/go/+/249880
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
@golang golang locked and limited conversation to collaborators Aug 22, 2021
@rsc rsc unassigned neild Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Projects
None yet
Milestone
Go1.15
Development

No branches or pull requests
5 participants
@neild @rsc @dmitshur @ianlancetaylor @gopherbot