Hey @JohnRusk! I understand that one of your concerns is with Azure/azure-storage-azcopy#113, in which the API is expecting capitalized headers, despite RFC2616 specifying that header fields are to be treated as case-insensitive.

Issue #5022 was resolved by preserving case on outbound requests:
f0396ca

I don't think it's reasonable to keep a mapping of every header field transformation around for every request. As @bradfitz mentioned, http/2 explicitly lowercases all fields. I feel like making changes here for broken servers (which I understand are out of your control) doesn't seem reasonable.

Is it possible to keep a map of these special header fields around in your application and re-write them on your outbound request? I assume there is a subset that are sensitive, and it is not all HTTP headers. Their casing is preserved on outbound requests as I understand #5022.

/cc @bradfitz @rsc

Is it possible to keep a map of these special header fields around in your application and re-write them on your outbound request?

Not really, because the set of header fields that exists is customer-determined, and may vary from customer to customer and even from file to file.

I don't think it's reasonable to keep a mapping of every header field transformation around for every request.

What about some kind of hook then, some kind of function that get's called as they are pulled of the wire, and before they are canonlicalized. If the hook is null, it doesn't get called.

e.g.(roughly)

headerName := ... <raw value off the wire>
if request.headerNameNotifier != nil {
   request.headerNameNotifier(headerName)
}
headerName := canonalicalize(headerName)

Then customers like us could provide request.headerNameNotifier and everyone else could leave it nil.

There are a few dup bugs in this space with a bunch of conversation. I can't find them at the moment (GitHub search is failing me) but they're somewhere.

I had trouble searching too. I have managed to find a few though. Are any of these the ones you were thinking of?

• net/http: allow control of case of header key #5022 . I see that there you proposed a workaround for writing (which has been implemented, albeit in a different form, if I understand correctly. But there's still no workaround for reading).

• net/http: CanonicalHeaderKey capitalization is non-conformant with RFC 6455 #18495 I notice there you wrote "I don't want to complicate Go and encourage buggy libraries from assuming case". I think the issue many of us have is that we have to interact with non-Go code that already assumes case, and assumes that we will preserve case (which we can't, when we're coding in Go). To be clear, no other library is assuming that we will be case sensitive, but they are assuming we will be case preserving when we handle their data.

• net/http/httputil: ReverseProxyHeader doesn't preserve header key case #18196

• net/textproto: allow manually register canonicalMIMEHeaderKey #22864

• Related : Canonical form of 'ETag' header is incorrect #18476

• Related, but not a dup: net/textproto: CanonicalMIMEHeaderKey ignores keys with _ in #13767

• Related, I think: net/textproto: ability to skip conversion for canonical format #29965

Is this https://azure.microsoft.com/en-in/updates/azure-data-factory-supports-preserving-metadata-during-file-copy/ a work around?

See also https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview.

@ACECEO yes, it's a workaround in the sense that it works. No in the sense that it's a totally different product, with a different emphasis and style of usage. For customers who use the tool I work on, AzCopy, this Go issue remains a blocker.

@bradfitz Any update re thoughts on this? As noted above, the key issue is that we currently can't write case-preserving code that reads headers and forwards them on. In our tool, AzCopy, this is a big deal for certain customers who need to move their data around in Azure.

Hi @ACECEO @toothrot, any additional thought?

RFC2616

I've read the RFC2616, it just mentions that the header is case-insensitive. Nothing says that you should modify the the headers as you forward them.

I feel like I've replied to this a number of places, but I'll summarize here as well.

• HTTP/2 makes this whole issue moot. So this bug only matters for HTTP/1<->HTTP/1 interactions, which becomes less relevant with each passing month.
• Code that's sensitive to HTTP/1 header case is already broken; catering towards such code implicitly encourages such brokenness or at least allows it to continue, without providing pressure on those applications to get fixed

As such, any fix here must have super minimal cost, both in terms of API surface (cogntitive load cost, reading more godoc) and runtime cost (we shouldn't allocate or populate a new data structure with this info unless the calling code opted in to wanting that). Notably, we can't change the representation of http.Header at this point.

It probably needs to be opt-in at the http.Server level. Perhaps a new bool there. That means "middleware" can't assume the original case is present, and can't ask for it to be present retroactively, as we'd only parse/preserve it if they opted in. So perhaps that's too inconvenient.

Maybe we just add a new RawHTTP1Header []byte field to http.Request, documenting that it's only for HTTP/1 and it's the exact bytes from the Request-Line to the final closing newlines. Then let code outside of the standard library deal with it. It'd also need to be documented that the slice is not a copy and it aliases some internal buffer and is only valid until the next request on the connection. That kinda sucks as an API requirement, but it's not too dissimilar to e.g. https://golang.org/pkg/bufio/#Reader.ReadSlice. And it sucks less than making all users pay for an allocation that very little code will ever use.

If somebody has an alternate API proposal or implementation I'll take a look, assuming it meets these general requirements.

Hi Brad. That suggestion about RawHTTP1Header sounds like a good option. Minimal impact, but enough to let us do what we need to. I presume the internal buffer, which it aliases, already exists. Is that right? If so, what about exposing it as method instead of a field? Perhaps called something like GetRawHTTP1Header() or CopyRawHttp1Header()? The method would make a copy from the internal buffer. But the copy would only happen if the method was called. So the only users who would pay the price of the allocation are those who actually need and use this feature. Perhaps that would be cleaner, in terms of API design, because the result of the method call is indeed a copy, and so there's no need to worry about the duration of its validity. (You just need to call the method before the next request is made on the connection).

Will it work client side too? (Just checking because you mention http.Server in the earlier paragraph, but we need it client-side, and I imagine other users will too.)

Perhaps called something like GetRawHTTP1Header() or CopyRawHttp1Header()? The method would make a copy from the internal buffer.

My reservation with a method is that they render pretty prominently in godoc HTML compared to, say, a struct field.

Perhaps that would be cleaner, in terms of API design, because the result of the method call is indeed a copy, and so there's no need to worry about the duration of its validity.

It would be cleaner, but OTOH this is definitely in the realm of "you should know what you're doing" territory so making a super comfortable & safe API isn't the highest priority.

BTW, the other constraint that would need to be documented on the lifetime of this slice is that it's only valid before the Request.Body is read from. Because that can also advance the buffer.

But now that I think about it more, I'm not even sure that this API works, as I think we only use a 2KB or 4KB bufio.Reader to read from the connection (so that's all we have in memory at a time) and we permit by default up to 1MB of headers (https://golang.org/pkg/net/http/#DefaultMaxHeaderBytes). So RawHTTP1Header []byte on by default won't work, at least in the general case. (And documenting that it only works for "small" HTTP requests seems pretty terrible.)

So we probably do need some sort of opt-in mechanism, and then it can't work in HTTP middleware packages that don't have control over the http.Server setup.

Will it work client side too?

Yeah, it probably should. I don't think we'd want to accept a change that only did one, lest it turn out that whatever API we pick isn't sufficient for the other. Seeing it work with both would be a good sign that the API was sufficient.

Good point re whether it should be documented prominently or not. (I agree with you that "not prominent" is preferable).

The question of whether it only works on small requests seems key. I think folks who have meaningful info in headers (such as some of our users) could easily have more than 4KB. Do you have any other suggestions? Shall we (our team here) see if we can come up with any alternative that meets the general requirements that you've outlined?

As I understand this problem, this prevents many users to just use traefik as a plugin-proxy, because it ALTERS the headers. There is a ton of software which is using case-sensitive headers and a transparent proxy should not change them...