New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
net/http: XML ContentLength for GET request differs between clients #19138
Comments
I modified your example a little bit so it behaves the way you expect. rest.DefaultTransport just dumps the raw request to stderr. package main
import (
"fmt"
"log"
"net/http"
"github.com/kevinburke/rest"
)
func main() {
examples := []string{
"http://www.xmlfiles.com/examples/note.xml",
//"https://www.gpo.gov/fdsys/bulkdata/BILLS/115/1/hconres/BILLS-115hconres18rds.xml",
//"https://www.w3schools.com/xml/cd_catalog.xml",
}
http.DefaultClient.Transport = &rest.Transport{Debug: true, Output: os.Stderr}
for i := range examples {
req, err := http.NewRequest("GET", examples[i], nil)
if err != nil {
log.Fatalln(err)
}
req.Header.Set("Accept-Encoding", "identity")
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatalln(err)
}
fmt.Println(resp.ContentLength)
resp.Body.Close()
}
for i := range examples {
resp, err := http.Head(examples[i])
if err != nil {
log.Fatalln(err)
}
fmt.Println(resp.ContentLength)
resp.Body.Close()
}
} Go is advertising it can accept gzip, the server sends back gzip, and Go unzips the response and strips the Content-Length header before returning the response body. HEAD requests do not advertise gzip because there's no body to unzip and HTTP/1.1 headers can't be compressed. It gets stripped here in net/http/transport.go: if rc.addedGzip && resp.Header.Get("Content-Encoding") == "gzip" {
resp.Body = &gzipReader{body: body}
resp.Header.Del("Content-Encoding")
resp.Header.Del("Content-Length")
resp.ContentLength = -1
resp.Uncompressed = true
} curl returns the "proper" value because it doesn't advertise unzipping by default, if you pass Accept-Encoding: gzip you notice the server Content-Length changes. |
see also the comment about "Uncompressed" on a net/http.Response: // Uncompressed reports whether the response was sent compressed but
// was decompressed by the http package. When true, reading from
// Body yields the uncompressed content instead of the compressed
// content actually set from the server, ContentLength is set to -1,
// and the "Content-Length" and "Content-Encoding" fields are deleted
// from the responseHeader. To get the original response from
// the server, set Transport.DisableCompression to true.
Uncompressed bool |
Ah I see. I don't know how I missed the Uncompressed flag in http.Response. Thank You! I'll make sure to check these issues with golang-nuts next time before filing an "issue." |
Please answer these questions before submitting your issue. Thanks!
What did you do?
Reading the ContentLength for an XML Response always returns -1. I'm not sure if this is intended behavior, since both curl and Chrome return the proper values. Changing the user-agent and cookie jar doesn't seem to affect the output, but sending a HEAD instead of GET returns the right value. Running this example outputs
What did you expect to see?
FWIW, I saw the same behavior in 1.7.4 as well.
System details
The text was updated successfully, but these errors were encountered: