Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: Default Client / HTTP mangles requests with specific long urls causing remote servers to throw 400s on valid requests #9543

Closed
nemosupremo opened this issue Jan 9, 2015 · 3 comments
Closed

net/http: Default Client / HTTP mangles requests with specific long urls causing remote servers to throw 400s on valid requests #9543

nemosupremo opened this issue Jan 9, 2015 · 3 comments
Labels
FrozenDueToAge

Comments

@nemosupremo
Copy link

1.) Go Version: go1.3 darwin/amd64 (Also affects go version go1.3 linux/amd64)
2.) OSX 10.9.5 / Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-24-generic x86_64)
3.)

When making a request with this specific a long url (it seems when the query_string is 266 characters), the go http client seems to mangle the request.

Here is a sample hits that hits the google homepage, making the query longer and longer until the error occurs: https://gist.github.com/nemothekid/501fb7e520999342560a

4.) For comparison, the python requests library doesn't have a problem with this:

>>> import requests
>>> r = requests.get("https://www.google.com?part=id&id=ZKfj0-ygXbg%2C8SINiaqgNIQ%2CxeQhXEJaHRo%2CU2VHAMOP5DY%2CGMr1JhKms5M%2CyKjYj5HMaNU%2CJgFAaxNc-tk%2CoMnDCxiZVMw%2Ck4VG725R4X0%2CU-s10nNV_N0%2CkN-p51dFYrk%2CmmIFr0NgRi8%2C4uuQ-qxjqbU%2CuRaePS8fjQE%2CJt_G6a4g9lw%2C5GaFxfjdfX8%2CoKoDg2aG-z4%2CrArN_38Xil8%2C -RuZqxSiFIw%2CNBvTI73iNDo%2CDaRvvkud9MQ%2CZm4y67kxpBk%2CG9MQIHE4pd0%2C2YwKlqgCdQw%2C6pc67X22kFo%2CYvxGDTawvAI%2CgpdFKiMGrDA%2CVEqtKfXYX6A%2ClqQCJ-xCEBE%2CmW8WwnqTmDA%2CZ0W_4kN8isA%2C3i8icj_wJGM%2C0i6E8FPecGc%2Cw8ISxSwSz-0%2CGx2ahhyKqEU%2CrFsJdFKsJDU%2C9ET_qEJYEew%2CVdXBiVM6hyI%2CefL0fPMZGCo%2CHH1OMx7dhhk%2ClvCd9zjXeZE%2CIJ0uQjwTc8o%2CMqoReZ6WJjU%2C0UkWtQGzdXc%2CIZjgPfWX8Jc%2CGvnULl2TRlw%2C6UNXAoOt-zk%2CQDaDKoGV-YA%2CX253vC7vvb8%2Cc-MWDL4C8PY&maxResults=50&")
>>> r.text
u'<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world\'s information, including webpages, images, videos and more. Google has many special features to help you find exactly what you\'re looking for." name="description"><meta content="noodp" name="robots"><meta content="/images/google_favicon_128.png" itemprop="image"><title>Google</title><script>(function(){window.google={kEI:\'tzGvVITFIomDoQTBrYGgCA...

5.) If you run the code above, Google (and other servers) you will get a 400 error. (I'm not sure what Google's 400 looks like, but I'm unsure even if the response completes).

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 400 (Bad Request)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/errors/logo_sm_2.png) no-repeat}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/errors/logo_sm_2_hr.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:55px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>400.</b> <ins>That’s an error.</ins>
  <p>Your client has issued a malformed or illegal request.  <ins>That’s all we know.</ins>

This doesn't effect every long url, for example in the gist provided if you set u to:

u := h + "?part=id&id=LxtN--HMNsA%2CgiIT5ALtjPY%2C5fmzALLsofw%2CahBMSnKmMHs%2C2lduGacyTTo%2CQTXrc-XrHc4%2ChleMcTD8qrk%2CLVa9IMK6-bg%2CF9SyVu74gWg%2Cz5ITVelLBYY%2Ccx-lqXEU_aI%2Cvcr4MzfQQvQ%2CCNBVPuHmn70%2C0VmHS9CQcZI%2CS7IDGVC1KN4%2CsuUOAusvDEc%2CMpg41_O5XOY%2CHa1a_UEIsiE%2CRLkWr_ybDqc%2C_YFGr_9vBwg%2CeIo9iuZ6REA%2CePCalOWbEn8%2CwYt_PSUtp-A%2CJzt6JODbssE%2ClfYHdaUfSjY%2CEtcGmHs3Qss%2CyaqNkw0HW7M%2CqxCfvgeeT58%2Cc2MSgxccv_0%2Cm8UnfePn2fg%2C4h0TiFKao2w%2CVo8yWORV7TI%2C2JxxLHafwMg%2CL_YTkZtfuAA%2CODV566R-4wc%2CPlZ4MQR0de4%2CzsbpRQDNQKQ%2CAGWNVxcF4P0%2CtgSAufCbf6M%2C1jXC6L4WREE%2C7GnhVg61l5w%2C2YEccc4Vp44%2CDg0E7pzAm8o%2CRKzWK1eyPAQ%2Ct-dPaxAuDkY%2Cjh3hRvNy1C0%2CtitAChBO0jc%2CBinjgB0bWjE%2Cx7rHmM-cUjs%2CttT0GmCLeVs&maxResults=50"

you can successfully execute that entire request with no problems. (I also don't know what the difference between the two urls, both are essentially random strings).
@minux minux closed this as completed Jan 9, 2015
@minux
Copy link
Member

minux commented Jan 9, 2015

There is a space at u[285], so when i = 287, the URL looks like this:
http://...../Xil8%2C -

And the GET request looks like this:
GET /?part=id&id=ZKfj0-....Xil8%2C - HTTP/1.1\r\n
...

Obviously an invalid GET request.

This is working as intended. You've passed an invalid URL.

@nemosupremo
Copy link
Author

Crap, sorry. I was looking at this so long I didn't see the obvious. Apologies for the unnecessary issue, and thanks for looking into this. (Although I still don't get why it worked in Python and in my browser)

@jayschwa
Copy link
Contributor

jayschwa commented Jan 9, 2015

Python and your browser might URL encode the space.

@golang golang locked and limited conversation to collaborators Jun 25, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@minux @jayschwa @nemosupremo @gopherbot