net/http: go 1.20.6 host validation breaks setting Host to a unix socket address #61431

nicks · 2023-07-19T00:27:07Z

What version of Go are you using (`go version`)?

$ go version
go version go1.20.6 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

can include this if requested

What did you do?

Create a request where the Host header is a unix socket address. Here's some runnable sample code:

package main

import (
	"context"
	"log"
	"net"
	"net/http"
	"os"
	"time"
)

func handler(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(204)
}

func main() {
	_ = os.Remove("/tmp/mysocket.sock")
	socket, err := net.Listen("unix", "/tmp/mysocket.sock")
	if err != nil {
		panic(err)
	}
	s := &http.Server{
		Handler:        http.HandlerFunc(handler),
	}

	go func() {
		c := &http.Client{
			Transport: &http.Transport{
				DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
					return net.Dial("unix", "/tmp/mysocket.sock")
				},
			},
		}
		req, _ := http.NewRequest("GET", "http://localhost.com", nil)
		req.Host = "unix:///tmp/mysocket.sock"
		resp, err := c.Do(req)
		if err != nil {
			log.Fatal(err)
		}
		log.Fatal("STATUS ", resp.Status)
	}()
	log.Fatal(s.Serve(socket))
}

What did you expect to see?

In Go 1.20.5, this prints:

2023/07/18 20:19:59 STATUS 204 No Content
exit status 1

What did you see instead?

In Go 1.20.6, this prints:

2023/07/18 20:20:29 Get "http://localhost.com": http: invalid Host header
exit status 1

Additional Info

Related discussion: #60374

My understanding is that this is an intentional change, to fix a security bug where the Host header contains newline characters. Here's the CVE: https://nvd.nist.gov/vuln/detail/CVE-2023-29406

Unforuntately, this also breaks CLIs in the Go ecosystem that set the Host header to a unix socket, for example : moby/moby#45935

Many projects silently upgrade from Go 1.20.5 -> 1.20.6, so we're starting to see this change break tons of projects in the go ecosystem.

My humble request is that the security fix on the Go 1.20 release branch could be more narrowly targeted at the security issue, and allow this Host header format, to unbreak the ecosystem. The Go 1.21 release line can more safely rollout the backwards-incompatible part of the change.

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2023-07-19T01:25:23Z

CC @neild @golang/security

rolandshoemaker · 2023-07-19T02:13:49Z

Not entirely sure how common this behavior is, but I will note that RFC 2616 Section 14.23 does seem to explicitly disallow this:

A client MUST include a Host header field in all HTTP/1.1 request
messages . If the requested URI does not include an Internet host
name for the service being requested, then the Host header field MUST
be given with an empty value.

seankhliao · 2023-07-19T03:43:56Z

This looks like misusing the Host header for what is arguably proxy/dialer functionality?

Aside: the file transport from net/http.NewFileTransport processes requests with URLs like file:///tmp/my.sock with an empty Host.

ChristianSch · 2023-07-19T11:39:30Z

I feel that discussing if this is allowed or not is not the most important aspect. As it worked before, it created an API and created expectations (see relevant xkcd). The fallout it created relates to a lot of docker-based technologies which make heavy use of sockets.

Examples I could find/affect me are:

k3d-io/k3d#1321
testcontainers/testcontainers-go#1359

As a lot of people are scrambling to fix the fallout, I think the good outcome will be a better understanding of potential downstream issues of „overcorrections“ and a better understanding of the standards in any case.

bcmills · 2023-07-19T14:08:29Z

Per #56986, the new validation should probably at least have a GODEBUG setting that allows (part or all of) the old behavior for incremental migration.

rolandshoemaker · 2023-07-19T16:43:25Z

It sounds like given the scope of those relying on this behavior we'll need to at least partial revert this. I'd like to keep the extremely strict behavior, at least behind a GODEBUG, so that we can work on pushing the ecosystem away from relying on this.

It seems like the main offender here is Docker?

cpuguy83 · 2023-07-19T16:53:03Z

It seems like the main offender here is Docker?

I would say any client connecting to an HTTP server over a unix socket.
So to answer the question, possibly given the breadth of use and the fact that it uses http with UDS.
But really Docker is just trying to set a meaningful value AND not setting req.Host does not work.

https://go.dev/play/p/JAHc0RFCMRy

neild · 2023-07-19T17:10:05Z

I don't think we need a GODEBUG; we can reduce the validation of outgoing Host headers to just checking that it's a valid header value, not that it's a valid Host header specifically. That's enough to ensure the outbound request is, at worst, something the server will reject.

gopherbot · 2023-07-19T17:34:19Z

Change https://go.dev/cl/511155 mentions this issue: net/http: restore truncation of Host headers after / or space

neild · 2023-07-19T17:39:38Z

Ugh, it's worse than I thought. Our previous behavior was to silently truncate Host headers in HTTP/1 requests at the first ' ' or '/' character.

I suspect Docker is relying on that silent truncation, since otherwise the invalid Host: unix:///tmp/mysocket.sock header is going to be rejected on the server. (The net/http server already validates the header, that hasn't changed recently.)

So I think the thing to do is reduce the validation to just checking that we're sending a valid header value (which is the minimum required) and restore the old truncation behavior. Perhaps we should have a GODEBUG for the truncation, although unless we have a plan for how to change it in the future that's probably just useless complexity.

gopherbot · 2023-07-19T17:47:02Z

Change https://go.dev/cl/511156 mentions this issue: http2: relax Host header validation

oleg-nenashev · 2023-07-20T15:15:09Z

Any chance something was backported to go1.19.11 linux/amd64 too? We started seeing a similar behavior there on GitHub actions: wiremock/wiremock-testcontainers-go#18

bcmills · 2023-07-20T15:17:55Z

@oleg-nenashev, yes, this change was backported to Go 1.19.11 as well. (See #61075.)

oleg-nenashev · 2023-07-20T15:23:15Z

@bcmills Yes, I confirm the regression: wiremock/wiremock-testcontainers-go#19

P.S: Probably wasn't the greatest time to release the new WireMock module for Testcontainers Go :) Will add to Errata while we we wait for a fix/revert on one of the sides

bcmills · 2023-07-20T20:09:26Z

Perhaps we should have a GODEBUG for the truncation, although unless we have a plan for how to change it in the future that's probably just useless complexity.

I suggest that:

We restore the truncation on the 1.19, 1.20, and 1.21 release branches, and
we fix the truncation behavior (and pass the full Host string as long as it is valid to send in a header) for Go 1.22, guarded by a GODEBUG and enabled by default only at Go 1.22 and above.

(The “Go 1.22 and above” part can be implemented by adding an entry to internal/godebugs/table.go with Changed: 22.)

cpuguy83 · 2023-07-20T20:14:41Z

That's my thought as well.
It was getting this on a patch release that really caused the pain.
We expect some things to break during major version updates.

Pin Go to version `1.20.5` to fix [testcontainer](https://golang.testcontainers.org/) tests. Go `1.20.6` introduced changes in `Host` header parsing ([issue](golang/go#61431)) which breaks `testcontainers-go` ([issue](testcontainers/testcontainers-go#1359)). Our tests started failing consistently yesterday due to this issue ([example run](https://github.com/xataio/pg-roll/actions/runs/5612153506)). The suggested workaround until this is resolved upstream is to pin to `1.20.5`.

mholt · 2023-07-28T18:40:26Z

If we are making HTTP requests to a unix socket, with this change, how are we supposed to stay compliant?

RFC 2616, Section 14.26:

A client MUST include a Host header field in all HTTP/1.1 request
messages. If the requested URI does not include an Internet host
name for the service being requested, then the Host header field MUST
be given with an empty value.

Since there is no Internet host name, the Host header "MUST" be given with an empty value.

Previously, " " (space) worked by sheer luck and allowed us to stay compliant.

Now if we set the Host header to a non-empty value, we are not only violating the RFC but we also make CORS validation & DNS rebinding mitigation tricky, since those checks require accurate hostnames, if any. So if we invent one like localhost or foo.local, we risk enabling an CORS breach or DNS rebinding attack.

I don't know that we can use an IP address like 127.0.0.1 or ::1 because we don't know which, if any, IP versions a host supports (hence their use of Unix sockets).

Anyone know if there's something I'm missing/overlooking about this patch so that we can still achieve our objective?

jub0bs · 2023-08-02T18:54:38Z

@mholt Not quite sure I'm following your comment about CORS, since the Host header plays no part in the CORS protocol.

mholt · 2023-08-02T19:02:52Z

DNS rebinding mitigation does, which is also important; but all the same, we need a host (even if it's used in the Origin header), if we cannot leave it blank.

Starting from Go 1.20.6 (net/http: go 1.20.6 host validation breaks setting Host to a unix socket address golang/go#61431) calling unix socket address no longer allows Host to be empty, Allow loopback hosts for admin endpoint from (caddyserver/caddy#5664)

neild · 2023-08-07T22:53:18Z

https://go.dev/cl/511155 changes the transport to send an empty Host header when the host is invalid, except in the case of a request sent to a proxy. Returning an error seems more useful than sending a destination-free request to a proxy. Sending an empty Host offers less potential for request smuggling than truncating at the first invalid character.

Are there any scenarios that I'm missing that this won't cover?

neild · 2023-08-07T22:53:37Z

@gopherbot please open backport issues. This is a regression.

gopherbot · 2023-08-07T22:53:50Z

Backport issue(s) opened: #61825 (for 1.19), #61826 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

gopherbot · 2023-08-11T17:52:58Z

Change https://go.dev/cl/518855 mentions this issue: [release-branch.go1.19] net/http: permit requests with invalid Host headers

gopherbot · 2023-08-11T17:53:23Z

Change https://go.dev/cl/518756 mentions this issue: [release-branch.go1.20] net/http: permit requests with invalid Host headers

gopherbot · 2023-08-11T17:54:11Z

Change https://go.dev/cl/518856 mentions this issue: [release-branch.go1.21] net/http: permit requests with invalid Host headers

* Adding support for pod auth * Bumped dependencies and exposed more pod auth fields * Tests now pass * Fix for bug golang/go#61431 * Checking the cluster certificate path * Updated the dependencies

…eaders Historically, the Transport has silently truncated invalid Host headers at the first '/' or ' ' character. CL 506996 changed this behavior to reject invalid Host headers entirely. Unfortunately, Docker appears to rely on the previous behavior. When sending a HTTP/1 request with an invalid Host, send an empty Host header. This is safer than truncation: If you care about the Host, then you should get the one you set; if you don't care, then an empty Host should be fine. Continue to fully validate Host headers sent to a proxy, since proxies generally can't productively forward requests without a Host. For #60374 Fixes #61431 Fixes #61904 Change-Id: If170c7dd860aa20eb58fe32990fc93af832742b6 Reviewed-on: https://go-review.googlesource.com/c/go/+/511155 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Roland Shoemaker <roland@golang.org> Run-TryBot: Damien Neil <dneil@google.com> (cherry picked from commit b9153f6) Reviewed-on: https://go-review.googlesource.com/c/go/+/518856 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Roland Shoemaker <roland@golang.org> Reviewed-by: Russ Cox <rsc@golang.org>

…eaders Historically, the Transport has silently truncated invalid Host headers at the first '/' or ' ' character. CL 506996 changed this behavior to reject invalid Host headers entirely. Unfortunately, Docker appears to rely on the previous behavior. When sending a HTTP/1 request with an invalid Host, send an empty Host header. This is safer than truncation: If you care about the Host, then you should get the one you set; if you don't care, then an empty Host should be fine. Continue to fully validate Host headers sent to a proxy, since proxies generally can't productively forward requests without a Host. For #60374 Fixes #61431 Fixes #61826 Change-Id: If170c7dd860aa20eb58fe32990fc93af832742b6 Reviewed-on: https://go-review.googlesource.com/c/go/+/511155 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Roland Shoemaker <roland@golang.org> Run-TryBot: Damien Neil <dneil@google.com> (cherry picked from commit b9153f6) Reviewed-on: https://go-review.googlesource.com/c/go/+/518756 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Russ Cox <rsc@golang.org> Run-TryBot: Roland Shoemaker <roland@golang.org>

…eaders Historically, the Transport has silently truncated invalid Host headers at the first '/' or ' ' character. CL 506996 changed this behavior to reject invalid Host headers entirely. Unfortunately, Docker appears to rely on the previous behavior. When sending a HTTP/1 request with an invalid Host, send an empty Host header. This is safer than truncation: If you care about the Host, then you should get the one you set; if you don't care, then an empty Host should be fine. Continue to fully validate Host headers sent to a proxy, since proxies generally can't productively forward requests without a Host. For #60374 Fixes #61431 Fixes #61825 Change-Id: If170c7dd860aa20eb58fe32990fc93af832742b6 Reviewed-on: https://go-review.googlesource.com/c/go/+/511155 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Roland Shoemaker <roland@golang.org> Run-TryBot: Damien Neil <dneil@google.com> (cherry picked from commit b9153f6) Reviewed-on: https://go-review.googlesource.com/c/go/+/518855 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Roland Shoemaker <roland@golang.org> Reviewed-by: Russ Cox <rsc@golang.org>

nicks mentioned this issue Jul 19, 2023

net/http: insufficient sanitization of Host header #60374

Closed

bcmills added the Unfortunate label Jul 19, 2023

bcmills added NeedsFix The path to resolution is known, but the work has not been done. Security labels Jul 19, 2023

bcmills added this to the Go1.21 milestone Jul 19, 2023

lostluck mentioned this issue Jul 19, 2023

[#27558] Update required Go version to 1.20. apache/beam#27559

Merged

3 tasks

srabraham mentioned this issue Jul 19, 2023

[Bug]: go 1.20.6: "http: invalid Host header, host port waiting failed" testcontainers/testcontainers-go#1359

Closed

Slijkhuis mentioned this issue Jul 20, 2023

fix: upgrade docker client testcontainers/testcontainers-go#1361

Closed

bojand mentioned this issue Jul 20, 2023

backend: pin verify action to go 1.20.5 redpanda-data/console#775

Merged

migmartri mentioned this issue Jul 20, 2023

testing containers not working on GitHub actions chainloop-dev/chainloop#255

Closed

oleg-nenashev mentioned this issue Jul 20, 2023

CI fails on go version go1.19.11 linux/amd64 wiremock/wiremock-testcontainers-go#18

Closed

bryan-aguilar mentioned this issue Jul 20, 2023

[chore] Reenable tests and linting disabled due to testconainters-go issue open-telemetry/opentelemetry-collector-contrib#24247

Merged

andrew-farries mentioned this issue Jul 21, 2023

Pin Go to 1.20.5 to work around testcontainers/Go breakage xataio/pgroll#40

Merged

mholt mentioned this issue Jul 31, 2023

core: Allow loopback hosts for admin endpoint (fix #5650) caddyserver/caddy#5664

Merged

This was referenced Aug 3, 2023

docker: error of docker exec -it ,return http: invalid Host header openwrt/packages#21667

Open

Compiling error: http: invalid Host header docker/cli#4437

Closed

usernamedt mentioned this issue Aug 7, 2023

Pin the CI golang version to 1.20.5 wal-g/wal-g#1525

Merged

gopherbot closed this as completed in b9153f6 Aug 7, 2023

This was referenced Aug 7, 2023

net/http: go 1.20.6 host validation breaks setting Host to a unix socket address [1.19 backport] #61825

Closed

net/http: go 1.20.6 host validation breaks setting Host to a unix socket address [1.20 backport] #61826

Closed

neild mentioned this issue Aug 9, 2023

net/http: go 1.20.6 host validation breaks setting Host to a unix socket address [1.21 backport] #61904

Closed

mcasperson added a commit to OctopusDeployLabs/terraform-provider-octopusdeploy that referenced this issue Aug 10, 2023

Fix for bug golang/go#61431

b223bf4

habuma mentioned this issue Aug 13, 2023

Homebrew installed Skaffold gives "invalid Host header" error GoogleContainerTools/skaffold#9019

Closed

dmitshur removed the Security label Aug 14, 2023

dmitshur modified the milestones: Go1.21, Go1.22 Aug 14, 2023

aciduck mentioned this issue Aug 17, 2023

runtime: crashes with "runtime: traceback stuck" #62086

Open

mfrw mentioned this issue Aug 18, 2023

golang: bump golang 1.19.12 -> 1.20.7 microsoft/azurelinux#6001

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: go 1.20.6 host validation breaks setting Host to a unix socket address #61431

net/http: go 1.20.6 host validation breaks setting Host to a unix socket address #61431

nicks commented Jul 19, 2023 •

edited

ianlancetaylor commented Jul 19, 2023

rolandshoemaker commented Jul 19, 2023

seankhliao commented Jul 19, 2023

ChristianSch commented Jul 19, 2023

bcmills commented Jul 19, 2023 •

edited

rolandshoemaker commented Jul 19, 2023

cpuguy83 commented Jul 19, 2023

neild commented Jul 19, 2023

gopherbot commented Jul 19, 2023

neild commented Jul 19, 2023

gopherbot commented Jul 19, 2023

oleg-nenashev commented Jul 20, 2023

bcmills commented Jul 20, 2023

oleg-nenashev commented Jul 20, 2023

bcmills commented Jul 20, 2023 •

edited

cpuguy83 commented Jul 20, 2023

mholt commented Jul 28, 2023

jub0bs commented Aug 2, 2023

mholt commented Aug 2, 2023

neild commented Aug 7, 2023

neild commented Aug 7, 2023

gopherbot commented Aug 7, 2023

gopherbot commented Aug 11, 2023

gopherbot commented Aug 11, 2023

gopherbot commented Aug 11, 2023

net/http: go 1.20.6 host validation breaks setting Host to a unix socket address #61431

net/http: go 1.20.6 host validation breaks setting Host to a unix socket address #61431

Comments

nicks commented Jul 19, 2023 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

Additional Info

ianlancetaylor commented Jul 19, 2023

rolandshoemaker commented Jul 19, 2023

seankhliao commented Jul 19, 2023

ChristianSch commented Jul 19, 2023

bcmills commented Jul 19, 2023 • edited

rolandshoemaker commented Jul 19, 2023

cpuguy83 commented Jul 19, 2023

neild commented Jul 19, 2023

gopherbot commented Jul 19, 2023

neild commented Jul 19, 2023

gopherbot commented Jul 19, 2023

oleg-nenashev commented Jul 20, 2023

bcmills commented Jul 20, 2023

oleg-nenashev commented Jul 20, 2023

bcmills commented Jul 20, 2023 • edited

cpuguy83 commented Jul 20, 2023

mholt commented Jul 28, 2023

jub0bs commented Aug 2, 2023

mholt commented Aug 2, 2023

neild commented Aug 7, 2023

neild commented Aug 7, 2023

gopherbot commented Aug 7, 2023

gopherbot commented Aug 11, 2023

gopherbot commented Aug 11, 2023

gopherbot commented Aug 11, 2023

nicks commented Jul 19, 2023 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

bcmills commented Jul 19, 2023 •

edited

bcmills commented Jul 20, 2023 •

edited