net/http: *Request.URL.Scheme returns an empty string. No alternative way present to get request url scheme. #28940

AnikHasibul · 2018-11-25T07:12:05Z

What version of Go are you using (`go version`)?

$ go version
go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

Yes!

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GOARCH="amd64"
GOBIN="/home/anix/go/bin"
GOCACHE="/home/anix/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/blackhole:/home/anix/gopath"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build796132705=/tmp/go-build -gno-record-gcc-switches"

What did you do?

package main

import (
        "fmt"
        "net/http"
)

func main() {
        fmt.Println("started!")
        http.HandleFunc("/", redr)
        http.ListenAndServe(":8080", nil)
}

func redr(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "SCHEME:", r.URL.Scheme, "HOST:", r.Host, "PATH", r.URL.Path, )
}

What did you expect to see?

$ curl -i http://localhost:8080/
HTTP/1.1 200 OK
Date: Sun, 25 Nov 2018 07:05:42 GMT
Content-Length: 36
Content-Type: text/plain; charset=utf-8

SCHEME: http  HOST: localhost:8080 PATH /

What did you see instead?

The SCHEME: is empty!

$ curl -i http://localhost:8080/
HTTP/1.1 200 OK
Date: Sun, 25 Nov 2018 07:05:42 GMT
Content-Length: 36
Content-Type: text/plain; charset=utf-8

SCHEME:  HOST: localhost:8080 PATH /

Yes!

I read this part of godoc!

$ go doc http.Request.URL

type Request struct {
    // URL specifies either the URI being requested (for server
    // requests) or the URL to access (for client requests).
    //
    // For server requests the URL is parsed from the URI
    // supplied on the Request-Line as stored in RequestURI. For
    // most requests, fields other than Path and RawQuery will be
    // empty. (See RFC 7230, Section 5.3)
    //
    // For client requests, the URL's Host specifies the server to
    // connect to, while the Request's Host field optionally
    // specifies the Host header value to send in the HTTP
    // request.
    URL *url.URL

    // ... other fields elided ...
}

The most focused part is:

// For server requests the URL is parsed from the URI
// supplied on the Request-Line as stored in RequestURI. For
// most requests, fields other than Path and RawQuery will be
// empty. (See RFC 7230, Section 5.3)

But there should be an alternative way to not get an empty string. As r.URL.Host has an alternative r.Host. But what about r.URL.Scheme?

No way!

The text was updated successfully, but these errors were encountered:

tcolgate · 2018-11-25T09:16:21Z

You won't get the scheme here. It is http by virtue of this being a plain (Non-TLS listener). A https request could not have worked.
If you had an extra TLS listener you need to determine which one got hit to infer the scheme, (You can look at the port used to connect, or just use an extra middleware on the TLS listener.

proglottis · 2018-11-26T02:08:17Z

I think checking this field for nil on http.Request is the closest you will get:

        // TLS allows HTTP servers and other software to record
        // information about the TLS connection on which the request
        // was received. This field is not filled in by ReadRequest.
        // The HTTP server in this package sets the field for
        // TLS-enabled connections before invoking a handler;
        // otherwise it leaves the field nil.
        // This field is ignored by the HTTP client.
        TLS *tls.ConnectionState

tcolgate · 2018-11-26T12:15:45Z

FWIW, since often what you will be wanting this for is to construct a URL non-relative URL for a client, it is desirable to try and respect the x-forwarded-proto header. The client may have used TLS, even though the server processing the request did not receive it over TLS.
@AnikHasibul I'd suggest closing this, it is not a bug.

andybons · 2018-11-26T18:39:06Z

Thanks.

Checking for a nil TLS field on an http.Request is the correct way to determine if a request is http. We likely will not provide an alternative method since it's redundant to do so given the current solution.

ChrisSalisbury · 2019-04-23T06:26:20Z

Checking TLS is all well and good if the only schemes in existence were http and https, but the real world is far broader than that. I arrived here looking for the best way to check for a ws or wss scheme for proxying purposes. I'm sure others have use cases for other schemes as well.

tcolgate · 2019-04-23T12:06:28Z

@ChrisSalisbury the scheme simply doesn't exist as part of the HTTP protocol. It's not in the request at all, there's no way to populate the field. The scheme is used when a client interprets a URL for the client to decide which language to use when talking to the server, there's no point where the client is mandate to tell the server what "language" is being used to talk to to it (there's an assumption that if we are communicating at all, then we must already know what language we are talking).

rzr · 2019-08-07T10:55:22Z

I am new to golang and I found the URL API very confusing,

IMHO, for developer convenience there should be an API somewhere (could be elsewhere than in http or url) to get the full URL from any http(s) request.

I sorted it out by re-constructing from above mentioned hints.

For the record here is a link to mentioned RFC:

https://tools.ietf.org/html/rfc7230#section-5.3

nicolasparada · 2019-08-23T17:33:03Z

Checking on the request URL TLS does the trick, thanks. But feels weird 😕
Checking the request URL scheme is where anyone would look at.

lthibault · 2019-09-24T11:23:04Z

The suggestions here focus on disambiguating http:// vs https://. How can we determine if an incoming request has e.g. a ws:// scheme?

tcolgate · 2019-09-24T11:35:54Z

The fundamental point here is that we simply do not know what the scheme is. Scheme is part of the URI spec. As unpalatable as it is, the http requests never gets the URL the client requested, it only gets the path (and, most of the time, the hostname),
Consider the case of a simple TCP based SSL proxy. The connection comes in as https, ssl is stripped, and the connection is forwarded on to the non-ssl server. Which scheme do you want? The one the client plugged into their browser (we can never get that), "https", because the original server saw a TLS connection, or "http".

DisposaBoy · 2019-09-24T11:47:57Z

@lthibault websockets start off as a normal HTTP(S) GET requests that come with headers that look like:

...
Upgrade: websocket
Connection: ...Upgrade...
...

lthibault · 2019-09-24T12:13:41Z

@DisposaBoy Right, but I was hoping to route requests at a given endpoint (e.g. /foo) towards a websocket handler or a vanilla HTTP handler based on the scheme of the incoming request.

Sounds like this isn't possible, so I'll likely have to use a query parameter such as ws://localhost:9021/foo?ws=1. This is ugly because the ws param is redundant with the ws scheme, but what can you do. ¯\_(ツ)_ /¯

tcolgate · 2019-09-24T12:28:07Z

@lthibault if you just want to know if it's a websocket request, can't you look for the connection upgrade headers?

DisposaBoy · 2019-09-24T12:39:32Z

@lthibault I'm not sure I understand. If you can detect that it's a WS connection by the presence of a ?ws=1 query I don't see why you can't detect it with the upgrade headers.

tcolgate · 2019-09-24T12:43:18Z

if you want to check ws: vs wss: then you can do both the check for an existing TLS negotiation (or X-Forwarded-For-Scheme), and the upgrade headers.

lthibault · 2019-09-24T13:28:31Z

@DisposaBoy @tcolgate Of course... how did I not think of that. 🤦‍♂

Many thanks!

andybons closed this as completed Nov 26, 2018

jmickey mentioned this issue Oct 18, 2019

Redirect to HTTPS oauth2-proxy/oauth2-proxy#285

Merged

3 tasks

valtlfelipe mentioned this issue Jan 23, 2020

Add http request scheme to logger traefik/traefik#6226

Merged

2 tasks

golang locked and limited conversation to collaborators Sep 23, 2020

gopherbot added the FrozenDueToAge label Sep 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: *Request.URL.Scheme returns an empty string. No alternative way present to get request url scheme. #28940

net/http: *Request.URL.Scheme returns an empty string. No alternative way present to get request url scheme. #28940

AnikHasibul commented Nov 25, 2018

tcolgate commented Nov 25, 2018 •

edited

proglottis commented Nov 26, 2018

tcolgate commented Nov 26, 2018

andybons commented Nov 26, 2018

ChrisSalisbury commented Apr 23, 2019

tcolgate commented Apr 23, 2019

rzr commented Aug 7, 2019 •

edited

nicolasparada commented Aug 23, 2019

lthibault commented Sep 24, 2019

tcolgate commented Sep 24, 2019

DisposaBoy commented Sep 24, 2019

lthibault commented Sep 24, 2019 •

edited

tcolgate commented Sep 24, 2019

DisposaBoy commented Sep 24, 2019 •

edited

tcolgate commented Sep 24, 2019

lthibault commented Sep 24, 2019

net/http: *Request.URL.Scheme returns an empty string. No alternative way present to get request url scheme. #28940

net/http: *Request.URL.Scheme returns an empty string. No alternative way present to get request url scheme. #28940

Comments

AnikHasibul commented Nov 25, 2018

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

tcolgate commented Nov 25, 2018 • edited

proglottis commented Nov 26, 2018

tcolgate commented Nov 26, 2018

andybons commented Nov 26, 2018

ChrisSalisbury commented Apr 23, 2019

tcolgate commented Apr 23, 2019

rzr commented Aug 7, 2019 • edited

nicolasparada commented Aug 23, 2019

lthibault commented Sep 24, 2019

tcolgate commented Sep 24, 2019

DisposaBoy commented Sep 24, 2019

lthibault commented Sep 24, 2019 • edited

tcolgate commented Sep 24, 2019

DisposaBoy commented Sep 24, 2019 • edited

tcolgate commented Sep 24, 2019

lthibault commented Sep 24, 2019

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

tcolgate commented Nov 25, 2018 •

edited

rzr commented Aug 7, 2019 •

edited

lthibault commented Sep 24, 2019 •

edited

DisposaBoy commented Sep 24, 2019 •

edited