Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/url: unexpected url encoding for ! in path #40894

Open
mh47838704 opened this issue Aug 19, 2020 · 3 comments
Open

net/url: unexpected url encoding for ! in path #40894

mh47838704 opened this issue Aug 19, 2020 · 3 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@mh47838704
Copy link

mh47838704 commented Aug 19, 2020 •
edited by davecheney

What version of Go are you using (go version)?

$ go version

Does this issue reproduce with the latest release?

alway Exist

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

What did you do?

func TestEncodedUrlPathEncode(t *testing.T) {
	urlString := "http://www.xyz.com/!^test.jpg"
	parsedUrl, _ := url.Parse(urlString)
	rawUrl := parsedUrl.EscapedPath()
	fmt.Println("encodedUrl：",rawUrl)
}

func TestEncodedUrlPathEncode(t *testing.T) {
	urlString := "http://www.xyz.com/!test.jpg"
	parsedUrl, _ := url.Parse(urlString)
	rawUrl := parsedUrl.EscapedPath()
	fmt.Println("encodedUrl：",rawUrl)
}

What did you expect to see?

for example

originUrl：https://github.com/!test.jpg
encodedUrl: https://github.com/!test.jpg    --- result is expected

originUrl: https://github.com/!^test.jpg
encodedUrl: https://github.com/%21%5Etest.jpg
expected: https://github.com/!%5eds.jpg

What did you see instead?

func (u *URL) EscapedPath() string {
	if u.RawPath != "" && validEncoded(u.RawPath, encodePath) {
		p, err := unescape(u.RawPath, encodePath)
		if err == nil && p == u.Path {
			return u.RawPath
		}
	}
	if u.Path == "*" {
		return "*" // don't escape (Issue 11202)
	}
	return escape(u.Path, encodePath)
}

i found the code int sdk, "validEncoded" method treat the "!" as encoded, but in "escape" define "!" need encoed when mode is “encodePath”， where is judged in method "shouldEscape", logic conflict, so the result is not expected.

and i found other language do not encode "!" "(" ")" , such as python
@davecheney
Copy link
Contributor

Looking at RFC 3986 ^ must always be percent encoded while ! may avoid percent encoding depending on its position in the URL. Section 3.3 indicates that ! need not be escaped. However any character in a path element may be percent escaped, for example https://google.com/a and https://google.com/%61 are identical.

I agree that the addition of a character that must be escaped into a path element triggering other sub-delims to be percent encoded is surprising, but by my reading of the RFC not wrong.

Can you explain how you found this issue and what problem is it causing for you?

@mh47838704
Copy link
Author

mh47838704 commented Aug 19, 2020 •
edited

i found this，cause i build a web proxy use go，a url through this proxy，the “!” has been encoded to "21%"，but the dest server also a proxy， dest server has a lot of rules to dispatch the request ，one of them is match the char “!”，because the "!" was encoded to "21%" when the request pass through my proxy server, so the dest server can not match correctly

Looking at RFC 3986 ^ must always be percent encoded while ! may avoid percent encoding depending on its position in the URL. Section 3.3 indicates that ! need not be escaped. However any character in a path element may be percent escaped, for example https://google.com/a and https://google.com/%61 are identical.

I agree that the addition of a character that must be escaped into a path element triggering other sub-delims to be percent encoded is surprising, but by my reading of the RFC not wrong.

Can you explain how you found this issue and what problem is it causing for you?

i found this，cause i build a web proxy use go，a url through this proxy，the “!” has been encoded to "21%"，but the dest server also a proxy， dest server has a lot of rules to dispatch the request ，one of them is match the char “!”，because the "!" was encoded to "21%" when the request pass through my proxy server, so the dest server can not match correctly

and i also found the encoded logic is not the same, when path only has char "!" and when the path has any other char that need encode, the output can not keep the same, this is not what i want

@cagedmantis cagedmantis changed the title url encode is unexpected net/url: unexpected url encode Aug 25, 2020
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 25, 2020
@cagedmantis cagedmantis added this to the Backlog milestone Aug 25, 2020
@cagedmantis
Copy link
Contributor

/cc @rsc @bradfitz

@seankhliao seankhliao changed the title net/url: unexpected url encode net/url: unexpected url encoding for ! in path Jul 16, 2022
@skonto skonto mentioned this issue Aug 28, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
3 participants
@davecheney @cagedmantis @mh47838704