I think this is happening in net/url.URL.setPath(). Line 674 unescapes lowercased encoding successfully, and line 679 escapes the unescaped value again which now produces uppercased encoding. Thus, the original lowercase encoded value, and the re-encoded uppercase value, although equivalent semantically, don't match (p == escp), resulting in RawPath being set.

This is testcase to show case-sensitive behavior. When unescaped character is uppercase in raw form RawPath will not be filled as escape will provide same value as p at if escp := escape(path, encodePath); p == escp {

func TestURL_ParseEscapingHexValues(t *testing.T) {
	var testCases = []struct {
		name          string
		when          string
		expectPath    string
		expectRawPath string
	}{
		{
			name:          "uppercase F in %3f",
			when:          "/ab%3Fde",
			expectPath:    "/ab?de",
			expectRawPath: `/ab%3Fde`, // actual RawPath is empty and test fails
		},
		{
			name:          "lowercase f in %3f",
			when:          "/ab%3fde",
			expectPath:    "/ab?de",
			expectRawPath: `/ab%3fde`,
		},
	}

	for _, tc := range testCases {
		t.Run(tc.name, func(t *testing.T) {
			u, _ := url.Parse(tc.when)

			if u.Path != tc.expectPath {
				t.Errorf("path `%s` is not `%s`", u.Path, tc.expectPath)
			}
			if u.RawPath != tc.expectRawPath {
				t.Errorf("RawPath `%s` is not `%s`", u.RawPath, tc.expectRawPath)
			}
		})
	}
}

Change https://go.dev/cl/417395 mentions this issue: net/url: normalize hex values before comparision

Use URL.EscapedPath to get the escaped path.

Parse only sets RawPath when the default escaped form of u.Path is different from the original path. When the two are the same, it leaves RawPath unset, since EscapedPath will reconstruct the original encoding.

See, for example: https://go.dev/play/p/TCNnJEl77Zb

This is not that useful when you compare performance difference between using RawPath and calling EscapedPath. If code for setting path knows how to deal with hex values it is perfectly normal to expect it to work with upper-case and lower-case hex values.

Parse only sets RawPath when the default escaped form of u.Path is different from the original path

I agree that "?" and "%3f" are different strings. But how are "%3f" and "%3F" different strings?

RFC 3986, Section 2.1 states that -

  The uppercase hexadecimal digits 'A' through 'F' are equivalent to
  the lowercase digits 'a' through 'f', respectively.  If two URIs
  differ only in the case of hexadecimal digits used in percent-encoded
  octets, they are equivalent.

So, according to RFC3986, "%3f" and "%3F" are equivalent.

RawPath is only set when the default escaped form of u.Path is different from the original path.

The way to get the escaped path of a url.URL is the EscapedPath method, which will consistently return the original path prior to parsing. See https://go.dev/play/p/TCNnJEl77Zb for an example: EscapedPath preserves the original path of both %3f and %3F.

RFC3986 §2.1

For consistency, URI producers and normalizers should use
uppercase hexadecimal digits for all percent-encodings.

IMO, the fact that u.EscapedPath() is returning a percent-encoded string that has lowercase hexadecimal digits is itself a bug.

EscapedPath preserves the original path of both %3f and %3F

IMO, it shouldn't. It should return %3F in both cases.

Or maybe we should keep EscapedPath unchanged (to not break applications that rely on this behaviour), and introduce a new function to compare two URLs?

@subnut

URI producers and normalizers should use
uppercase hexadecimal digits for all percent-encodings.

From its documentation, EscapedPath returns u.RawPath when it is a valid escaping of u.Path. Otherwise it computes an escaped form on its own. Only in this latter case it can be considered an URI producer.

But note that the RFC uses the word should and not must, and in a RFC this difference is relevant. So I think this behavior shouldn't be considered a bug anyway.

IMO, the fact that u.EscapedPath() is returning a percent-encoded string that has lowercase hexadecimal digits is itself a bug.

For a URL constructed by url.Parse, EscapedPath returns the original path, preserving the exact escaping used in that original path.

For a URL constructed manually, such as url.URL{Path: "a?b"}, EscapedPath uses upper case hexadecimal digits to escape the path as recommended by RFC 3986.

Guys, you do understand that we are dealing with things that clients send over the (inter)net.

RawPath value being different if path contains %3f or %3F is unexpected even so that both Path are unescaped fine but somehow RawPath result is empty/filled for one case.

https://go.dev/play/p/566KcdtC18-

func main() {
	lowerCase, _ := url.Parse("/ab%3fde")
	fmt.Printf("RawPath for /ab%%3fde is \"%s\" and Path is \"%s\"\n", lowerCase.RawPath, lowerCase.Path)

	upperCase, _ := url.Parse("/ab%3Fde")
	fmt.Printf("RawPath for /ab%%3Fde is \"%s\" and Path is \"%s\"\n", upperCase.RawPath, upperCase.Path)
}

creates output:

RawPath for /ab%3fde is "/ab%3fde" and Path is "/ab?de"
RawPath for /ab%3Fde is "" and Path is "/ab?de"

how is this consistent behavior?

@aldas I think you are giving RawPath a meaning it doesn't have. It is not meant to represent the encoded path. It exists only and exclusively as a support to the EscapedPath method.

The documentation says encoded path hint (see EscapedPath method) and the commit message 874a605 that added RawPath is more explicit: Clients should use EscapedPath instead of reading RawPath directly.

@neild Would it be better if the RawPath documentation was rewritten as encoded path hint (use EscapedPath instead of reading it directly)?

From docs I see and agree that RawPath is to differentiate if you had / (unescaped) or %2f (escaped character) but I really do not find anything about it being case sensitive for hex chars in escaped characters.

even docs have lower case hex value and no mentions that %2f %2F would be different in any way

So it should be?

 // - setPath("/foo/bar")   will set Path="/foo/bar" and RawPath="" 
 // - setPath("/foo%2Fbar") will set Path="/foo/bar" and RawPath="" 
 // - setPath("/foo%2fbar") will set Path="/foo/bar" and RawPath="/foo%2fbar" 

this %2f %2F differentiation looks so bizarre.

The documentation of the URL type says

... RawPath, an optional field which only gets set if the default encoding is different from Path.

where with default we should mean the encoding used by the PathEscape function.

This documentation is wrong, it should be

... RawPath, an optional field which only gets set if the default encoding of Path is different from the original path.

that is consistent with the setPath documentation.

@aldas In light of this, your previous example

RawPath for /ab%3fde is "/ab%3fde" and Path is "/ab?de"
RawPath for /ab%3Fde is "" and Path is "/ab?de"

behaves correctly and explains the "bizarre" behavior.

Also this example is correct

RawPath for /ab%2fde is "/ab%2fde" and Path is "/ab/de"
RawPath for /ab%2Fde is "/ab%2Fde" and Path is "/ab/de"

If the original path is /ab%2Fde, Path is /ab/de and its default encoding is /ab/de. Because /ab/de is different from
/ab%2Fde, RawPath is set to /ab%2Fde and not to an empty string.

Could someone explain me if I develop HTTP server related code - request router or "middleware" or http handler and I need to get actual path part that client sent, where can I get it without doing extra work? Real world example - I want to serve file downloads where part of path is the file name and therefore I can not use Path because it is being unescaped automatically.

func handler(w http.ResponseWriter, r *http.Request) {
	io.WriteString(w, fmt.Sprintf("raw path part: %s\n", r.URL.RawPath))
}

func main() {
	http.HandleFunc("/", handler)
	if err := http.ListenAndServe(":8080", nil); err != http.ErrServerClosed {
		log.Print(err)
	}
}

"most" of the time RawPath works but when we are dealing with request containing escaped values somehow the case of escaped hex value matter.

so for curl http://localhost:8080/file/report%3fpart1.csv I can get hold of "raw"/actual part what was sent
but for curl http://localhost:8080/file/report%3Fpart1.csv I can not - unless I do the extra work (that was already done in setPath) and call EscapedPath().

Why do we need to call EscapedPath() for uppercase curl http://localhost:8080/file/report%3Fpart1.csv and do escape again inside EscapedPath(), the same work twice as setPath called also escape on same path? This just baffles me.

I am not native English speaker - so to summarize:
Why do upper-case hex value %2F requires twice the amount of work to get HTTP request "raw path part" that client actually sent?

Could someone explain me if I develop HTTP server related code - request router or "middleware" or http handler and I need to get actual path part that client sent, where can I get it without doing extra work?

http.Request.RequestURI

This is going nowhere. http.Request.RequestURI contains query and fragments parts and forces you to parse it to get path part. Now you are still doing extra work like you would do with EscapePath but atleast you can now if you are dealing with upper/lower case hex value.

I assume %2f %2F are so completely different things and case matters for when comes to URL that is sourced from internetz.

So at least docs should explictly state in public API that these %2f %2F are different.

For example setPath method has useful comment about RawPath behaviour but setPath is private method so you probably will not see that

It would be useful if URL comment would explicitly state that case matters for escaped values. Currently this is not obvious from

// This distinction is rarely important,
// but when it is, the code should use RawPath,

this part does not take into consideration that for %2F RawPath is empty and you have no way knowing if URL was escaped or not. It would be useful if there would be example for 3 cases like that:

 // - "/foo/bar"   will set Path="/foo/bar" and RawPath="" 
 // - "/foo%2Fbar" will set Path="/foo/bar" and RawPath="" 
 // - "/foo%2fbar" will set Path="/foo/bar" and RawPath="/foo%2fbar" 

RawPath comment is not that useful as it circular reference with EscapePath and does not help you with hidden cotchas. I personally would think that it would be helpful if there would be displaimer that Raw does not mean "pure" in this field context and you can not assume part of original string that was parsed into URL when it comes to hex digits.

and EscapePath() comment

EscapedPath returns u.RawPath when it is a valid escaping of u.Path

does this sentence means that path that contains %2F was not valid?

Now take these strings for example and see what happens when you send escaped value

https://go.dev/play/p/LNKg1upv9ZP

func main() {
	for _, p := range []string{
		"report.csv%3fpart1?a=1", // lowercase ?
		"report.csv%3Fpart1?a=1", // uppercase ? <-- RawPath does not exists
		"%5e",                    // lowercase ^
		"%5E",                    // uppercase ^ <-- RawPath does not exists
		"%7ebashrc",              // lowercase ~
		"%7Ebashrc",              // uppercase ~ <-- RawPath exists
	} {
		u, err := url.Parse(p)
		if err != nil {
			fmt.Println(err)
			continue
		}
		fmt.Printf("original: %s\t, Path: %s, EscapedPath(): %s\t,RawPath: %s\n", p, u.Path, u.EscapedPath(), u.RawPath)
	}
}

Output

original: report.csv%3fpart1?a=1	, Path: report.csv?part1, EscapedPath(): report.csv%3fpart1	,RawPath: report.csv%3fpart1
original: report.csv%3Fpart1?a=1	, Path: report.csv?part1, EscapedPath(): report.csv%3Fpart1	,RawPath: 
original: %5e	, Path: ^, EscapedPath(): %5e	,RawPath: %5e
original: %5E	, Path: ^, EscapedPath(): %5E	,RawPath: 
original: %7ebashrc	, Path: ~bashrc, EscapedPath(): %7ebashrc	,RawPath: %7ebashrc
original: %7Ebashrc	, Path: ~bashrc, EscapedPath(): %7Ebashrc	,RawPath: %7Ebashrc

somehow now RawPath does not care about hexadecimal digits case for ~ and is filled.

This is not intuitive at all.

This is commit that added RawPath 874a605

In that commit message @rsc wrote

Split the difference by treating u.RawPath as a hint: the observation
is that there are many valid encodings of u.Path. If u.RawPath is one
of them, use it.

Does "many valid encodings" mean that 3f and 3F are both valid or not? By my understanding of escaping and https://datatracker.ietf.org/doc/html/rfc3986#section-2.1 stating

The uppercase hexadecimal digits 'A' through 'F' are equivalent to
the lowercase digits 'a' through 'f', respectively.

would it mean that RawPath should get filled for both cases? To hint that original path was encoded.

At the moment cases like %3F or %5E this knowledge is lost but RawPath exists to hint that and for performance reasons this is very useful. Because if URL was not encoded you can/could opt out of calling EscapedPath or doing extra work to parse actual raw/original path from http.Request.RequestURI

and this question still remains:

Why do upper-case hex values like %3F %5E requires twice the amount of work to get HTTP request "raw path part" that client actually sent? setPath calls escape once and not when you are forced to use EscapedPath you are calling escape twice.

Change https://go.dev/cl/418035 mentions this issue: net/url: clarify RawPath documentation

The original implementation of URL did not preserve the original unescaped form of a URL. After parsing a URL string, there was no way to distinguish between a/b, a%2fb, and a%2Fb, since these all escape to the same string. The escaped form of Path was accessible via the EscapedPath method.

Go 1.15 changed the EscapedPath method to return the original path. It did this by adding the RawPath field, which is set only when the escaped path is different from the default escaping of Path.

I think there's an argument that RawPath should have been an unexported field, to avoid confusion, but it's far too late to worry about that now. The behavior of RawPath is fairly straightforward, however:

• Code which parses raw URLs will set RawPath if the escaped path is different from Path.
• EscapedPath will return RawPath if it contains a valid escaping of Path.

The RawPath field is not present for efficiency purposes. It exists to allow EscapedPath to return the original encoded form of a URL. Most code should use EscapedPath rather than accessing RawPath directly.