Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: client will only send a single user-agent header value #41738

Closed
mdwhatcott opened this issue Oct 1, 2020 · 4 comments
Closed

net/http: client will only send a single user-agent header value #41738

mdwhatcott opened this issue Oct 1, 2020 · 4 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@mdwhatcott
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.15.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/mike/Library/Caches/go-build"
GOENV="/Users/mike/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/mike/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/mike"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mv/rt57xj7n1xl1wvqn5h0y2x4m0000gp/T/go-build718939165=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

request, _ := http.NewRequest("GET", "http://localhost:8080/", nil)
request.Header.Add("Content-Type", "A")
request.Header.Add("Content-Type", "B")
request.Header.Add("User-Agent", "C")
request.Header.Add("User-Agent", "D")
_, _ = http.DefaultClient.Do(request)

What did you expect to see?

I expected the server to receive two Content-Type values ("A" & "B") and two User-Agent values ("C" & "D").

What did you see instead?

The server did receive two Content-Type values as expected, but only the first User-Agent value ("C") was received.

Is this a bug or oversight, or is there a good reason for the Go HTTP client to not send multiple user-agent values?


The http.Header type models HTTP headers, which can appear multiple times in the same request/response, by using a map[string][]string (note the []string value). The following snippet demonstrates that some headers, such as User-Agent are only sent once by the Go HTTP client in an outgoing HTTP request:

package main

import (
	"log"
	"net/http"
)

func main() {
	log.SetFlags(0)

	go func() { _ = http.ListenAndServe("localhost:8080", new(echo)) }()

	request, _ := http.NewRequest("GET", "http://localhost:8080/", nil)

	request.Header.Add("Content-Type", "A")
	request.Header.Add("Content-Type", "B")
	request.Header.Add("User-Agent", "C")
	request.Header.Add("User-Agent", "D") // Will not be sent!

	log.Println("Client:", request.Header)

	_, _ = http.DefaultClient.Do(request)
}

type echo struct{}

func (echo) ServeHTTP(_ http.ResponseWriter, request *http.Request) {
	request.Header.Del("Accept-Encoding")
	log.Println("Server:", request.Header)
}

The output of the above program is as follows:

Client: map[Content-Type:[A B] User-Agent:[C D]]
Server: map[Content-Type:[A B] User-Agent:[C]]

However, using other tools (such as cURL) it is possible to send multiple User-Agent headers:

$ curl -v "https://www.google.com" -H 'User-Agent: 1' -H 'User-Agent: 2'
...
> GET / HTTP/2
> Host: www.google.com
> Accept: */*
> User-Agent: 1
> User-Agent: 2
...

Why does Go disallow multiple User-Agent values?

For reference, I believe this is the code that limits the User-Agent to a single value:

https://github.com/golang/go/blob/master/src/net/http/h2_bundle.go#L8025-L8028

} else if strings.EqualFold(k, "user-agent") {
	// Match Go's http1 behavior: at most one
	// User-Agent. If set to nil or empty string,
	// then omit it. Otherwise if not mentioned,
	// include the default (below).
	didUA = true
	if len(vv) < 1 {
		continue
	}
	vv = vv[:1]
	if vv[0] == "" {
		continue
	}
}

Is this a bug or oversight, or is there a good reason for the Go HTTP client to not send multiple user-agent values?

@andybons
Copy link
Member

andybons commented Oct 1, 2020

@bradfitz @neild

@andybons andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 1, 2020
@andybons andybons added this to the Unplanned milestone Oct 1, 2020
@bradfitz
Copy link
Contributor

bradfitz commented Oct 1, 2020

Looks pretty intentional at least (per that comment).

Also seems like reasonable behavior. Certain well-known headers only appear once.

Arguably Content-Type could be capped to just 1 too, for security reasons.

@fraenkel
Copy link
Contributor

fraenkel commented Oct 2, 2020

See https://tools.ietf.org/html/rfc7230#section-3.2.2
Sending duplicate headers is only valid if the values are comma separated.

@mdwhatcott
Copy link
Author

@fraenkel - Thanks for pointing that out! I had been looking for some pronouncement like that but wasn't able to see it.

@golang golang locked and limited conversation to collaborators Oct 2, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants