Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/textproto: CanonicalMIMEHeaderKey ignores keys with _ in #13767

Closed
ncw opened this issue Dec 29, 2015 · 4 comments
Closed

net/textproto: CanonicalMIMEHeaderKey ignores keys with _ in #13767

ncw opened this issue Dec 29, 2015 · 4 comments

Comments

@ncw
Copy link
Contributor

ncw commented Dec 29, 2015

textproto.CanonicalMIMEHeaderKey ignores keys with _ in. If an htttp server returns one of these, then it isn't possible to use http.Header.Get to fetch it as it doesn't get canonicalized.

For example, Backblaze use keys with _ in to store metadata - see the X-Bz-Info-src_last_modified_millis in the upload docs.

The Backblaze server returns this header all in lower case. Because it doesn't get canonicalised by textproto.CanonicalMIMEHeaderKey it remains in lower case and consequently `resp.Header.Get("X-Bz-Info-src_last_modified_millis") doesn't find it.

This is easy enough to work around, but caused me a bit of suprise and debugging!

The comment on textproto/reader.go on validHeaderFieldByte explains the problem nicely!

// validHeaderFieldByte reports whether b is a valid byte in a header
// field key. This is actually stricter than RFC 7230, which says:
//   tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
//           "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA
//   token = 1*tchar
// TODO: revisit in Go 1.6+ and possibly expand this. But note that many
// servers have historically dropped '_' to prevent ambiguities when mapping
// to CGI environment variables.
func validHeaderFieldByte(b byte) bool {
    return ('A' <= b && b <= 'Z') ||
        ('a' <= b && b <= 'z') ||
        ('0' <= b && b <= '9') ||
        b == '-'
}

This could cause problems with S3 also which uses a similar scheme for returning user defined metadata too.

@bradfitz
Copy link
Contributor

Oh, sorry, this slipped through everybody's cracks. We've all been so focused on the Go 1.6 dashboard and nobody triaged this for a few days when it came in (oh, Dec 29th), dooming it.

I made a demo here: http://play.golang.org/p/aGv7ijAOx_

In summary, CanonicalMIMEHeaderKey is currently basically this (simplifed):

    func CanonicalMIMEHeaderKey(s string) {
        if strings.Contains(s, "_") { // or other bogus stuff
             return s
        }
        // else actually canonicalize it....
        ...
    }

I think we should probably add underscore as an allowed byte. We'd just not mess with the case of the following the byte like we do for hyphen.

/cc @rsc

Maybe this can still be Go 1.6 material?

@gopherbot
Copy link

CL https://golang.org/cl/18725 mentions this issue.

@bradfitz bradfitz added this to the Go1.7Early milestone Jan 20, 2016
@bradfitz
Copy link
Contributor

Actually, I can just imagine this breaking too many tests. Let's submit at the beginning of Go 1.7 instead.

@ncw
Copy link
Contributor Author

ncw commented Jan 20, 2016

@bradfitz thanks for looking at this - Go 1.7 sounds fine to me.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants