Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http.Get(url) returns error 503 but web page loads on browser just fine #16343

Closed
Aukstkalnis opened this issue Jul 13, 2016 · 3 comments
Closed

Comments

@Aukstkalnis
Copy link

  1. What version of Go are you using (go version)?
    1.6
  2. What operating system and processor architecture are you using (go env)?
    windows 7
  3. What did you do?
    My code :

package main
import (
"fmt"
"log"
"net/http"
)

var addr = "http://www.wuxiaworld.com/category/tdg-chapter-release"

func PeriodicUpdate() {
response, err := http.Get(addr)
if err != nil {
log.Fatal(err)
} else {
defer response.Body.Close()
fmt.Print(response)
}
}
func main() {
PeriodicUpdate()
}

  1. What did you expect to see?
    Full body of the page
  2. What did you see instead?
    &{503 Service Temporarily Unavailable 503 HTTP/1.1 1 1 map[Date:[Wed, 13 Jul 2016 07:49:26 GMT] Content-Type:[text/html; charset=UTF-8]
    Refresh:[8;URL=/cdn-cgi/l/chk_jschl?pass=1468396170.809-WlFrmyimVA] Cf-Ray:[2c1b236a86711577-FRA] Set-Cookie:[__cfduid=da9936617d2fd099e
    19daa7d006ae49111468396166; expires=Thu, 13-Jul-17 07:49:26 GMT; path=/; domain=.wuxiaworld.com; HttpOnly] X-Frame-Options:[SAMEORIGIN]
    Cache-Control:[no-cache] Server:[cloudflare-nginx]] 0x10b360c0 -1 [chunked] true map[] 0x10af4c40 }

In browser this page works OK. So why does it not work in GO?

@acoshift
Copy link

I think that website is protected by cloudflare.

$ curl -I http://www.wuxiaworld.com/category/tdg-chapter-release
HTTP/1.1 503 Service Temporarily Unavailable
Date: Wed, 13 Jul 2016 07:59:49 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Set-Cookie: __cfduid=d523b2957b624fc265a4cf98cebbf18e91468396789; expires=Thu, 13-Jul-17 07:59:49 GMT; path=/; domain=.wuxiaworld.com; HttpOnly
X-Frame-Options: SAMEORIGIN
Refresh: 8;URL=/cdn-cgi/l/chk_jschl?pass=1468396793.743-51J/2UqXLx
Cache-Control: no-cache
Server: cloudflare-nginx
CF-RAY: 2c1b329fee320811-SIN

@odeke-em
Copy link
Member

That page is protected by Cloudflare DDOS protection, ie the page first checks the browser and user-agent before it accepts accessing the page.
screen shot 2016-07-13 at 1 57 42 am

Here is the actual body parsed also from Go, I've modified your original code to dump out the response's body.

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
)

var addr = "http://www.wuxiaworld.com/category/tdg-chapter-release"

func PeriodicUpdate() {
    req, err := http.NewRequest("GET", addr, nil)
    if err != nil {
        log.Fatal(err)
    }

    res, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Fatal(err)
    }

    defer res.Body.Close()
    slurp, _ := ioutil.ReadAll(res.Body)
    fmt.Printf("%v\n%v\n%s\n", res, res.Request, slurp)
}
func main() {
    PeriodicUpdate()
}
$ go run 16343.go
&{503 Service Temporarily Unavailable 503 HTTP/1.1 1 1 map[Content-Type:[text/html; charset=UTF-8] Set-Cookie:[__cfduid=d70d9a2f056b736b6815426a609b3fa571468397422; expires=Thu, 13-Jul-17 08:10:22 GMT; path=/; domain=.wuxiaworld.com; HttpOnly] X-Frame-Options:[SAMEORIGIN] Refresh:[8;URL=/cdn-cgi/l/chk_jschl?pass=1468397426.153-gUc3d7y750] Server:[cloudflare-nginx] Date:[Wed, 13 Jul 2016 08:10:22 GMT] Cf-Ray:[2c1b421071c5500b-DEN] Cache-Control:[no-cache]] 0xc420012540 -1 [chunked] true false map[] 0xc4200d20f0 <nil>}
&{GET http://www.wuxiaworld.com/category/tdg-chapter-release HTTP/1.1 1 1 map[] <nil> 0 [] false www.wuxiaworld.com map[] map[] <nil> map[]   <nil> <nil> <nil> <nil>}
<!DOCTYPE HTML>
<html lang="en-US">
<head>
  <meta charset="UTF-8" />
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  <meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
  <meta name="robots" content="noindex, nofollow" />
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
  <title>Just a moment...</title>
  <style type="text/css">
    html, body {width: 100%; height: 100%; margin: 0; padding: 0;}
    body {background-color: #ffffff; font-family: Helvetica, Arial, sans-serif; font-size: 100%;}
    h1 {font-size: 1.5em; color: #404040; text-align: center;}
    p {font-size: 1em; color: #404040; text-align: center; margin: 10px 0 0 0;}
    #spinner {margin: 0 auto 30px auto; display: block;}
    .attribution {margin-top: 20px;}
    @-webkit-keyframes bubbles { 33%: { -webkit-transform: translateY(10px); transform: translateY(10px); } 66% { -webkit-transform: translateY(-10px); transform: translateY(-10px); } 100% { -webkit-transform: translateY(0); transform: translateY(0); } }
    @keyframes bubbles { 33%: { -webkit-transform: translateY(10px); transform: translateY(10px); } 66% { -webkit-transform: translateY(-10px); transform: translateY(-10px); } 100% { -webkit-transform: translateY(0); transform: translateY(0); } }
    .bubbles { background-color: #404040; width:15px; height: 15px; margin:2px; border-radius:100%; -webkit-animation:bubbles 0.6s 0.07s infinite ease-in-out; animation:bubbles 0.6s 0.07s infinite ease-in-out; -webkit-animation-fill-mode:both; animation-fill-mode:both; display:inline-block; }
  </style>

    <script type="text/javascript">
  //<![CDATA[
  (function(){
    var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
    b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
    b(function(){
      var a = document.getElementById('cf-content');a.style.display = 'block';
      setTimeout(function(){
        var s,t,o,p,b,r,e,a,k,i,n,g,f, rmUDWhW={"VMNIZLbMLiR":!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]};
        t = document.createElement('div');
        t.innerHTML="<a href='/'>x</a>";
        t = t.firstChild.href;r = t.match(/https?:\/\//)[0];
        t = t.substr(r.length); t = t.substr(0,t.length-1);
        a = document.getElementById('jschl-answer');
        f = document.getElementById('challenge-form');
        ;rmUDWhW.VMNIZLbMLiR-=+((!+[]+!![]+[])+(!+[]+!![]+!![]+!![]));rmUDWhW.VMNIZLbMLiR-=+((!+[]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));rmUDWhW.VMNIZLbMLiR*=+((+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));rmUDWhW.VMNIZLbMLiR*=+((!+[]+!![]+[])+(+[]));rmUDWhW.VMNIZLbMLiR*=+((!+[]+!![]+!![]+[])+(!+[]+!![]));rmUDWhW.VMNIZLbMLiR+=+((!+[]+!![]+[])+(+!![]));rmUDWhW.VMNIZLbMLiR+=+((+!![]+[])+(+!![]));a.value = parseInt(rmUDWhW.VMNIZLbMLiR, 10) + t.length; '; 121'
        f.submit();
      }, 4000);
    }, false);
  })();
  //]]>
</script>


</head>
<body>
  <table width="100%" height="100%" cellpadding="20">
    <tr>
      <td align="center" valign="middle">
          <div class="cf-browser-verification cf-im-under-attack">
  <noscript><h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the page.</h1></noscript>
  <div id="cf-content" style="display:none">
    <div>
      <div class="bubbles"></div>
      <div class="bubbles"></div>
      <div class="bubbles"></div>
    </div>
    <h1><span data-translate="checking_browser">Checking your browser before accessing</span> wuxiaworld.com.</h1>
    <p data-translate="process_is_automatic">This process is automatic. Your browser will redirect to your requested content shortly.</p>
    <p data-translate="allow_5_secs">Please allow up to 5 seconds&hellip;</p>
  </div>
  <form id="challenge-form" action="/cdn-cgi/l/chk_jschl" method="get">
    <input type="hidden" name="jschl_vc" value="cfc957550467040d96be99f96d314f35"/>
    <input type="hidden" name="pass" value="1468397426.153-gUc3d7y750"/>
    <input type="hidden" id="jschl-answer" name="jschl_answer"/>
  </form>
</div>


          <div class="attribution">
            <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=iuam" target="_blank" style="font-size: 12px;">DDoS protection by CloudFlare</a>
            <br>
            Ray ID: 2c1b421071c5500b
          </div>
      </td>
    </tr>
  </table>
</body>
</html>

You'll probably need some sort of headless browser to wait until the number of seconds of wait has expired and also to accept the automatic redirect

@Aukstkalnis
Copy link
Author

It works :) Thank you :)

@golang golang locked and limited conversation to collaborators Jul 13, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants