net/http: Transport does not handle 302 redirects with link-local server address correctly #20669

max2k1 · 2017-06-14T13:51:19Z

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (`go version`)?

go version go1.8.3 linux/amd64

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr/local/go"
GORACE=""
GOROOT="/opt/go"
GOTOOLDIR="/opt/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

I need to get some data from link-local neighbor:

package main

import (
	"fmt"
	"net/http"
)

func main() {
	if resp, err := http.Get("http://[fe80::1%25eth0]:8002/info.json"); err == nil {
		defer resp.Body.Close()
		fmt.Println(resp)
	} else {
		fmt.Println(err)
	}
}

What did you expect to see?

Something like this:

&{200 OK 200 HTTP/1.1 1 1 map[Connection:[keep-alive] Set-Cookie:[uid=AAAAAVlBPd2KJ2cFAwMUAg==; path=/] Server:[nginx] Date:[Wed, 14 Jun 2017 13:45:01 GMT] Content-Type:[application/json]] 0xc42007a600 -1 [chunked] false false map[] 0xc4200e8300 <nil>}

What did you see instead?

Get http://[fe80::1]:8002/info.sh: dial tcp [fe80::1]:8002: connect: invalid argument

It happened, because http://[fe80::1%eth0]:8002/info.json returned 302-redirect to /info.sh and http-library lost zone-prefix somewhere while following redirect.
Proof:

curl -v "http://\[fe80::1%25eth0\]:8002/info.json"
*   Trying fe80::1...
* Connected to fe80::1 (fe80::1) port 8002 (#0)
> GET /info.json HTTP/1.1
> Host: [fe80::1]:8002
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 302 Moved Temporarily
< Server: nginx
< Date: Wed, 14 Jun 2017 13:49:17 GMT
< Content-Type: text/html
< Content-Length: 154
< Location: http://[fe80::1]:8002/info.sh
< Connection: keep-alive
< Set-Cookie: uid=AAAAAVlBPt2KJ2cFAwMVAg==; path=/
<
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<center><h1>302 Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host fe80::1 left intact

The text was updated successfully, but these errors were encountered:

bradfitz · 2017-06-14T14:56:33Z

Where is this behavior defined in a spec?

That 302 response's Location header is Location: http://[fe80::1]:8002/info.sh and doesn't have a zone. Are we really supposed to keep state and associate a zone with an authority for some duration?

bradfitz · 2017-06-14T14:56:38Z

/cc @mikioh

max2k1 · 2017-06-14T17:11:51Z

Where is this behavior defined in a spec?
That 302 response's Location header is Location: http://[fe80::1]:8002/info.sh and doesn't have a zone.

It seems to me, there are no RFCs concerning this behaviour. Web server just use requst's HOST-header, we've send to it, in response's Location-header, it has no idea about zones, because they're local to client. It is client, who should decide, how to reach specified location.

bradfitz · 2017-06-14T17:24:19Z

I don't want to complicate Go's code for a niche use case that's not even defined.

max2k1 · 2017-06-14T17:32:04Z

But it just does not work now: strange exception after correct http-server response... It's weird. Modern curl, for example, deals with that kind of addresses in responses easily.

bradfitz · 2017-06-14T17:41:31Z

Curl is probably not handling it and just handling connection cache management differently and you're getting lucky.

If you find evidence that curl is explicitly handling this in some way, then I might be convinced to do the same.

Please go grep the curl source code and figure out whether curl deals with IPv6 zones.

mikioh · 2017-06-14T17:51:51Z

See https://tools.ietf.org/html/rfc6874. Handling "beyond routing scope" is one of classic IPv6 issues and that's the reason why IPv6 site-local unicast addresses are deprecated: https://tools.ietf.org/html/rfc3879.

So using IPv6 literals in HTTP has some restriction. I'm guessing that one control plane solution could be the combo of a) using mDNS for propagating on-link adjacency information and b) using non-IPv6 literals, node/service names propagated by mDNS, in HTTP.

max2k1 · 2017-06-14T18:01:43Z

that's the reason why IPv6 site-local unicast addresses are deprecated: https://tools.ietf.org/html/rfc3879.

fe80::1 is not kind of site-local, but a link-local address – they will not be deprecated in the near future for sure :). It is like an 192.168.0.1 in ipv4 world, because of a common pattern to set it on default gw devices and route through it. How often do you see http://192.168.0.1/something URL's? I think you see them often enough. Now try to imagine, that you favourite language http-transport does not handle 302 to 192.168.0.1 address...

bradfitz · 2017-06-14T18:05:12Z

@max2k1, I think you messed up your quoting there. I'm not sure how to read that comment.

Somebody needs to translate this into English for me. I speak a little networking, but not as a native language.

max2k1 · 2017-06-14T18:13:28Z

Curl is probably not handling it and just handling connection cache management differently and you're getting lucky.

Curl does recognise different scopes of ip6-addresses here: https://github.com/curl/curl/blob/93567071b744975e191c5732d9cbe4b0d529f4c9/lib/if2ip.c#L69-L76
and selects correct interface there: https://github.com/curl/curl/blob/93567071b744975e191c5732d9cbe4b0d529f4c9/lib/if2ip.c#L115-L193

mikioh · 2017-06-14T18:25:17Z

translate this into English

See https://tools.ietf.org/html/rfc6874#section-4. RFC 3879 describes the rest of pains on operating "scope boundaries."

It is like an 192.168.0.1 in ipv4 world

Nope, see https://tools.ietf.org/html/rfc3927. The equivalent of IPv4 private addresses is https://tools.ietf.org/html/rfc4193.

max2k1 · 2017-06-14T18:34:32Z

@max2k1, I think you messed up your quoting there. I'm not sure how to read that comment.
Somebody needs to translate this into English for me. I speak a little networking, but not as a native language.

Sorry, I'll try to be simpler.
fe80::1 is a link-local address. Usually, you do not need global addresses in ipv6 world to communicate over one shared link (ethernet segment for example). Instead, you got automatically generated link-local addresses on every interface of you network station (fe80::/64) and they can be used to reach any of your neighbour. So every network interface of any ipv6-enabled device always have link-local address (one, or more).
It's a common practice in ipv6-enabled networks to use link-local addresses (fe80::1 usually for "default gateways") as routers' addresses, because it is very handy. That is why I compared fe80::1 with 192.168.0.1. 192.168.0.1 is widely used as default gateway address and, because of that, frequently used in http-requests. So, in IPv6-only world it is common to query over http link-local addresses. That is why I think, that it is important to fully support this case in golang http-transport.

bradfitz · 2017-06-14T18:41:21Z

And why doesn't it work now?

Per RFC 6874, we already remove the zone identifier from the outgoing Host header: https://github.com/golang/go/blob/release-branch.go1.8/src/net/http/request.go#L678

That was added in 957255f and has tests.

So we already can successfully make requests to http://[fe80::1%whatever]:80.

But what says that when that server replies with a redirect code + Location header response without a zone, which zone are we supposed to use?

max2k1 · 2017-06-14T18:46:56Z

But what says that when that server replies with a redirect code + Location header response without a zone, which zone are we supposed to use?

We know local interface of this tcp-connection. So this is zone id.

bradfitz · 2017-06-14T18:50:29Z

Okay, I think I finally understand what you want.

In the future, please write your bug report with all the information.

I would consider a patch for this if it's minimally invasive, but I don't intend to work on this myself.

mikioh · 2017-06-15T01:26:25Z

I have no strong opinion on this feature because I have no measurement for IPv6 addressing/routing scope boundary handling in transport- and/or session-layer protocols, such as HTTP.

FWIW, the net/http package provides the CheckRedirect field of http.Client for such use cases.

SCKelemen · 2017-07-06T03:03:59Z

@SCKelemen

gopherbot · 2017-09-20T08:45:29Z

Change https://golang.org/cl/64911 mentions this issue: net/http: remove ipv6 link-local address zone ID from connection cache key

crvv · 2017-09-20T09:11:13Z

This looks like an undefined behavior of HTTP, so I think supporting this explicitly is not necessary.
Other implementations (I checked cURL and Python) don't support this explicitly.

cURL extracts zone ID from link-local address and store them separately. cURL uses both of them to setup connections and use address as key to cache connections. So the second request sent by cURL uses the previous connection and can send request successfully.

Python doesn't handle zone ID at all.

The relevant cURL code is
https://github.com/curl/curl/blob/8839c05fba1f8415eec17eff8ac60cc3a50eb51e/lib/url.c#L4749
https://github.com/curl/curl/blob/8839c05fba1f8415eec17eff8ac60cc3a50eb51e/lib/conncache.c#L123

I think using the same logic(do not use zone ID in cache key) in Go is enough.

mikioh · 2018-04-26T09:32:10Z

FYI: https://lists.w3.org/Archives/Public/ietf-http-wg/2018JanMar/0067.html

bradfitz changed the title ~~http.Get does not handle 302 redirects with link-local server address correctly~~ net/http: Transport does not handle 302 redirects with link-local server address correctly Jun 14, 2017

bradfitz added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jun 14, 2017

bradfitz added this to the Go1.10 milestone Jun 14, 2017

bradfitz modified the milestones: Unplanned, Go1.10 Jun 14, 2017

mikioh mentioned this issue Mar 6, 2019

net/url: Parse documentation does not adequately explain escaping rules or RFC compliance #30611

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: Transport does not handle 302 redirects with link-local server address correctly #20669

net/http: Transport does not handle 302 redirects with link-local server address correctly #20669

max2k1 commented Jun 14, 2017

bradfitz commented Jun 14, 2017

bradfitz commented Jun 14, 2017

max2k1 commented Jun 14, 2017 •

edited

bradfitz commented Jun 14, 2017

max2k1 commented Jun 14, 2017

bradfitz commented Jun 14, 2017

mikioh commented Jun 14, 2017

max2k1 commented Jun 14, 2017 •

edited

bradfitz commented Jun 14, 2017

max2k1 commented Jun 14, 2017

mikioh commented Jun 14, 2017 •

edited

max2k1 commented Jun 14, 2017 •

edited

bradfitz commented Jun 14, 2017

max2k1 commented Jun 14, 2017

bradfitz commented Jun 14, 2017

mikioh commented Jun 15, 2017 •

edited

SCKelemen commented Jul 6, 2017

gopherbot commented Sep 20, 2017

crvv commented Sep 20, 2017 •

edited

mikioh commented Apr 26, 2018

net/http: Transport does not handle 302 redirects with link-local server address correctly #20669

net/http: Transport does not handle 302 redirects with link-local server address correctly #20669

Comments

max2k1 commented Jun 14, 2017

What version of Go are you using (go version)?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

bradfitz commented Jun 14, 2017

bradfitz commented Jun 14, 2017

max2k1 commented Jun 14, 2017 • edited

bradfitz commented Jun 14, 2017

max2k1 commented Jun 14, 2017

bradfitz commented Jun 14, 2017

mikioh commented Jun 14, 2017

max2k1 commented Jun 14, 2017 • edited

bradfitz commented Jun 14, 2017

max2k1 commented Jun 14, 2017

mikioh commented Jun 14, 2017 • edited

max2k1 commented Jun 14, 2017 • edited

bradfitz commented Jun 14, 2017

max2k1 commented Jun 14, 2017

bradfitz commented Jun 14, 2017

mikioh commented Jun 15, 2017 • edited

SCKelemen commented Jul 6, 2017

gopherbot commented Sep 20, 2017

crvv commented Sep 20, 2017 • edited

mikioh commented Apr 26, 2018

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

max2k1 commented Jun 14, 2017 •

edited

max2k1 commented Jun 14, 2017 •

edited

mikioh commented Jun 14, 2017 •

edited

max2k1 commented Jun 14, 2017 •

edited

mikioh commented Jun 15, 2017 •

edited

crvv commented Sep 20, 2017 •

edited