net/http: Transport doesn't support deflate (and Twitter is broken) #18779
Comments
|
Go is telling Twitter "Accept-Encoding: gzip", and Twitter is replying "Content-Encoding: deflate". Seems like a bug on their side. See http://stackoverflow.com/questions/388595/why-use-deflate-instead-of-gzip-for-text-files-served-by-apache for why gzip is preferred over deflate. Not sure there's anything for us to do here. I suppose we could attempt to un-deflate things. /cc @tombergan @dsnet |
bradfitz
changed the title
|
This is definitely a bug on twitter's end; we asked for "gzip", and they gave us "deflate". I don't think we should transparently decompress things with the "deflate" TE since there is some confusion about what the actual format of "deflate" is. RFC 2616 specifies that "deflate" should actually be the zlib format (RFC 1950), but there are some implementations that accidentally treat it as raw DEFLATE (RFC 1951). If we tried to transparently decompress, we could run into decompression errors because of the wrong format. Worse yet, neither raw DEFLATE or zlib have any magic values, so you cannot differentiate it from the zlib format. The internet ended up avoiding "deflate" as a TE and moved to "gzip" to avoid this confusion. I wouldn't want the Go implementation to get mixed up in all that confusion. |
|
Copying https://github.com/twitter/netty-http2 folk who can maybe help: @yschimke, @atollena Twitter appears to be returning "deflate"-compressed responses when a client only asks for "gzip". |
If we want to language-lawyer RFC 7231, I believe that servers are technically not required to obey Accept-Encoding; that header is just a recommendation. For example, note the use of SHOULD instead of MUST: https://tools.ietf.org/html/rfc7231#section-5.3.4
That said, I would still consider this a bug in Twitter's server ... any other interpretation invites insanity. Some clients will decode any response with a known Content-Encoding, independent of whether or not the encoding was explicitly allowed via the request's Accept-Encoding. One such client is Chrome. It would not be entirely unreasonable for a Go program to do the same, however, I probably would not bake that behavior into the standard library.
I think you're saying that the meaning of "Content-Encoding: deflate" is ambiguous, due to the existence of some broken implementations? This problem seems orthogonal to the Accept-Encoding issue, and in any case, I'd be inclined to consider an implementation broken if it does not follow RFC 2616. The updated text from RFC 7230/7234 agrees with the old text from RFC 2616: |
|
Cannot reproduce: |
|
Ah, I wrote too soon. Let me try with http/2. Getting a curl which can do http/2, will report back soon. |
|
As far as I can tell, on http/2, twitter.com always gzips, and blog.twitter.com always deflates. Seems odd to me, I'll poke around tomorrow. |
|
I worked around this twitter bug 6 months ago. Apologies, I should have actually worked with you guys to fix it. |
|
@mosesn, thanks! |
Twitter returns deflate data despite the client only requesting gzip data. net/http automatically handles the latter but not the former: golang/go#18779 Fixes ChimeraCoder#170.
Twitter returns deflate data despite the client only requesting gzip data. net/http automatically handles the latter but not the former: golang/go#18779 Fixes ChimeraCoder#170.
Twitter returns deflate data despite the client only requesting gzip data. net/http automatically handles the latter but not the former: golang/go#18779 Fixes ChimeraCoder#170.
Twitter returns deflate data despite the client only requesting gzip data. net/http automatically handles the latter but not the former: golang/go#18779 Fixes ChimeraCoder#170.
Twitter returns deflate data despite the client only requesting gzip data. net/http automatically handles the latter but not the former: golang/go#18779 Fixes #170.
|
This is still happening but it still seems like it must be Twitter's fault, unless HTTP/2 requires all clients to support deflate decoding. Moving to Go 1.11. |
|
@rsc see my comment above, Twitter may fix it, but it is a known twitter bug you will probably need to workaround. |
|
@rsc @bradfitz this is in flight, targeted for early January. sorry for the long delay, it fell off my plate for a bit. https://twittercommunity.com/t/improving-the-twitter-api-support-for-http-2/98728 |
|
@mosesn, thanks for the fix and update! Will close. |
What version of Go are you using (
go version)?go version go1.7.4 linux/amd64What operating system and processor architecture are you using (
go env)?What did you do?
https://play.golang.org/p/ArzIv0uBlQ
What did you expect to see?
I see binary pre-decompression crap.
net/http doc says "If the Transport requests gzip on its own and gets a gzipped response, it's transparently decoded in the Response.Body. However, if the user explicitly requested gzip it is not automatically uncompressed".
I'm not requesting it, so it's requested on its own.
What did you see instead?
I expect it to be decompressed, since I didn't request compression.
I don't know if this is because blog.twitter.com does something out of spec.
The text was updated successfully, but these errors were encountered: