Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: net/http/httputil: can ReverseProxy copy src header to dst header with Set() instead of Add() ? #50730

Closed
LawyZheng opened this issue Jan 21, 2022 · 14 comments

Comments

@LawyZheng
Copy link

I use Gin Web Framework to code my HTTP Server,
and handling some requests with ReverseProxy.

I registered the CORS in the Gin Middleware before the requests are handled by the ReverseProxy.
However, unfortunately, my ReverseProxy target server also implements the CORS headers,
which results in the CORS headers becoming duplicated,
and causing the response to be refused by the browser which said it got two Access-Control-Allow-Origin in the headers.
I also came across the same problem when I used the Gin Gzip Middleware in a similar situation.

So I just looked into the source code and found it use the Add() function to copy the headers,

func copyHeader(dst, src http.Header) {
for k, vv := range src {
for _, v := range vv {
dst.Add(k, v)
}
}
}

Maybe it is better to use the Set() function which can replace the existing values?
Or at least I think we need an option to use Add() or Set()?

@gopherbot gopherbot added this to the Proposal milestone Jan 21, 2022
@LawyZheng
Copy link
Author

And I have added the HeadersOverwrite Option by myself.
I'm glad to create a PR if needed.

@ianlancetaylor
Copy link
Contributor

CC @neild

@ianlancetaylor ianlancetaylor changed the title proposal: proposal: net/http/httputil: Can ReverseProxy copy src header to dst header with Set() instead of Add() ? proposal: net/http/httputil: can ReverseProxy copy src header to dst header with Set() instead of Add() ? Jan 21, 2022
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Jan 21, 2022
@LawyZheng
Copy link
Author

I add a custom way to modify headers,
and this is my commits.
I will create a PR if welcomed.

master...LawyZheng:master

@rsc
Copy link
Contributor

rsc commented Feb 9, 2022

It sounds like maybe this is a duplicate of the more general problem in #50465?
And #50465 is pretty intractable at the moment unfortunately.
We probably need a whole new proxy API.

@rsc rsc moved this from Incoming to Active in Proposals (old) Feb 9, 2022
@rsc
Copy link
Contributor

rsc commented Feb 9, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@LawyZheng
Copy link
Author

In my understanding, I don't think it is a duplicate problem in #50465.
#50465 seems to talk about the headers in the request to track where the request is coming from,
while this issue is trying to talk about the headers in the response.

@chrisguiney
Copy link
Contributor

http allows for headers to be specified multiple times for headers that can be combined with a ,

from rfc 2616

Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded.

For example, duplicate headers are often used for X-Forwarded-For instead of combing them into a comma separated list.

In a response, a good example of using a header multiple times is to set multiple cookies with Set-Cookie, in which mozilla advises:

To send multiple cookies, multiple Set-Cookie headers should be sent in the same response.

@LawyZheng
Copy link
Author

http allows for headers to be specified multiple times for headers that can be combined with a ,

from rfc 2616

Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded.

For example, duplicate headers are often used for X-Forwarded-For instead of combing them into a comma separated list.

In a response, a good example of using a header multiple times is to set multiple cookies with Set-Cookie, in which mozilla advises:

To send multiple cookies, multiple Set-Cookie headers should be sent in the same response.

Yes, I can't agree with you anymore.
Maybe I didn't explain my proposal clearly in the title and content,
I don't mean that we should use replace instead of append,
I'd like to say that we are able to choose the method we want when we use ReverseProxy.

@rsc
Copy link
Contributor

rsc commented Mar 30, 2022

It seems clear that ReverseProxy is not right in certain contexts, but we don't know the right path forward for it.

Perhaps the right next step is to put this on hold (or decline it) and suggest that people fork ReverseProxy and experiment in their own copies?

@chrisguiney
Copy link
Contributor

Maybe I didn't explain my proposal clearly in the title and content

I think it's I who did not read your proposal accurately

I personally have no objection to the patch supplied, but I am curious why it can't be handled in ModifyResponse(*http.Response)? -- for example, to remove headers that you don't want propagated. It looks like headers get copied only after ModifyResponse has been called. t's probably less error prone and more ergonomic to think in terms of what headers to copy, versus what to remove so it isn't copied.

Another thought: I personally find myself reaching for httptest.ResponseRecorder, or similar http.ResponseWriter implementations to do things like capture upstream response codes, headers, or body.

I think if the standard library had a simple http.ResponseRecorder that didn't make some of the choices that httptest.ResponseRecorder does, then a lot of situations like this wouldn't be as painful, because they could easily have a middleware in front of them to capture/rewrite headers as necessary.

That's not a perfect solution either, because it likely means that the reverseproxy response needs to be buffered, which may be undesirable, especially for larger requests.

@LawyZheng
Copy link
Author

for example, to remove headers that you don't want propagated. It looks like headers get copied only after ModifyResponse has been called.

Actually, I've solved this problem in this way now. But I don't think it's a good solution, because it feels like solving this problem as a special case in certain special requests, rather than solving it from a project common perspective.

For example, I need to make sure three things (the middleware, ReverseProxy, response headers which I care about) are compatible with each other to handle the request correctly. From my personal view, I don't think it's a good way for project maintenance.

As you mentioned, many frameworks will not buffer the HTTP body considering the performance, which means the middleware will write the headers before ReverseProxy does. So maybe it's better to implement it in the ReverseProxy struct?

Just my personal opinion.

@rsc
Copy link
Contributor

rsc commented Apr 6, 2022

It seems like we should decline this until we have a clearer idea of a path forward (or a new package).

@rsc
Copy link
Contributor

rsc commented Apr 13, 2022

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

@rsc rsc moved this from Active to Likely Decline in Proposals (old) Apr 13, 2022
@rsc rsc moved this from Likely Decline to Declined in Proposals (old) May 4, 2022
@rsc
Copy link
Contributor

rsc commented May 4, 2022

No change in consensus, so declined.
— rsc for the proposal review group

@rsc rsc closed this as completed May 4, 2022
@golang golang locked and limited conversation to collaborators May 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests

5 participants