Comment 1:

Thanks.  This will be easy to fix once we have the new Dial in place.

Owner changed to @bradfitz.

Status changed to Accepted.

Comment 2 by Paul.A.Lalonde:

I totally botched the repro conditions - I should have let it sit over lunch.
It is, of course, only when you open the connection from Go, using Get or equivalent,
that the proxies get invoked, which is where my app is having grief.  I'll add a
reproducer shortly.

Comment 3 by Paul.A.Lalonde:

Here's a quick modification of pkg/http/proxy_test.go that at least includes some cases
I care about.  I chose "stable" external addresses to lookup.
A question: is the new Dial a 1 week horizon, or a multi-month horizon?  For the later
I'll throw some extra logic into my user code; for the former I'll grin and bear it :-)
Thanks,
    Paul

Attachments:

1. proxy_test.go (1720 bytes)

Comment 4:

probably closer to the latter.

Comment 5:

Now that we have ParseCIDR, this is less work than it was before:
http://tip.goneat.org/pkg/net/#IP.ParseCIDR
But what are the rules for the "HTTP_PROXY" environment variable?  Is there a spec of
any sort?
Do we really have to do the hostname DNS lookup first before deciding whether or not to
use a proxy?  That is, if "foo.intranet" resolves to 10.2.3.4, it should trigger for
10.0.0.0/8?
I just want to do whatever other browsers do.

Status changed to WaitingForReply.

Comment 6 by Paul.A.Lalonde:

http://www.w3.org/Daemon/User/Proxies/ProxyClients.html looks like the most
authoritative resource.
Paul

Comment 7:

Well, I think we comply with that spec already:
"Some clients support the no_proxy environment variable that specifies a set of domains
for which the proxy should not be consulted; the contents is a comma-separated list of
domain names, with an optional :port part:
        no_proxy="cern.ch,ncsa.uiuc.edu,some.host:8080"
    export no_proxy
"
Let's try to find something written past July 1995.  :)

Comment 8 by Paul.A.Lalonde:

I should have added that "authoritative" doesn't include "complete".  
no_proxy including subnets/CIDR notation is something I got from Chrome, I think.
Paul

Comment 9:

[+evan@chromium.org, Chrome guy who writes Go and might have answers]

Comment 10:

I went to check Chrome's code to see what no_proxy meant, but my code search turned up
Python and Perl's code first.  Maybe you'll find all of them helpful:
http://codesearch.google.com/codesearch#search/&exact_package=chromium&q=no_proxy&type=cs
Here's the relevant Chrome code:
http://codesearch.google.com/codesearch#OAMlx_jo-ck/src/net/proxy/proxy_config_service_linux.cc&exact_package=chromium&ct=rc&cd=13&q=no_proxy&sq=&l=179
It appears there is code for special semantics of no_proxy.
I don't have enough of a checkout at the moment to look at the blame.
PS. Proxy settings are a huge complex mess.  People always yell at us for not using
libproxy which knows how to e.g. obey gconf and evaluate JavaScript for PAC files.

Comment 11:

Hey, you know what would make any HTTP client better?
Needing to evaluate arbitrary JavaScript code to decide
how to connect to a host.  :-)
Russ

Comment 12:

I knew there was a reason for the Go V8 bindings.  But I think we need to put Nigel onto
a pure Go port for the standard library.  :)

Comment 13:

Labels changed: added priority-later.

Comment 14:

Status changed to Accepted.

Comment 15:

Labels changed: added go1.1.

Comment 16:

Labels changed: added size-m.

Comment 17:

Labels changed: added suggested.

Comment 18:

I haven't started working on this (and would love if somebody else wants to do this),
but to get some requirements here on the bug:
a) do we want to do a DNS lookup on the hostname in order to get an IP address to match
against?  (I believe that's what this bug is about)
b) behavior if more than 1 IP address?  Just pick a random one and assume they're all in
the same class?
c) do we care about caching the DNS lookup so it's not done again by Transport's dialer
in the non-proxied case, or do we trust the system resolver to cache?

Comment 19:

Looks to me like we only care if the URL uses an IP address.
That is, if you say no_proxy=10.2.3.4 and foo.com resolves to 10.2.3.4, http://10.2.3.4/
is no_proxy but http://foo.com/ is not (is proxied).
From Chrome:
http://code.google.com/p/chromium/source/search?q=BypassIPBlockRule%5C+%3A%5C+public&origq=BypassIPBlockRule%5C+%3A%5C+public&btnG=Search+Trunk
That's proxy_bypass_rules.cc:
class BypassIPBlockRule : public ProxyBypassRules::Rule {
 public:
  // |ip_prefix| + |prefix_length| define the IP block to match.
  BypassIPBlockRule(const std::string& description,
                    const std::string& optional_scheme,
                    const IPAddressNumber& ip_prefix,
                    size_t prefix_length_in_bits)
      : description_(description),
        optional_scheme_(optional_scheme),
        ip_prefix_(ip_prefix),
        prefix_length_in_bits_(prefix_length_in_bits) {
  }
  virtual bool Matches(const GURL& url) const OVERRIDE {
    if (!url.HostIsIPAddress())
      return false;
    if (!optional_scheme_.empty() && url.scheme() != optional_scheme_)
      return false;  // Didn't match scheme expectation.
    // Parse the input IP literal to a number.
    IPAddressNumber ip_number;
    if (!ParseIPLiteralToNumber(url.HostNoBrackets(), &ip_number))
      return false;
    // Test if it has the expected prefix.
    return IPNumberMatchesPrefix(ip_number, ip_prefix_,
                                 prefix_length_in_bits_);
  }
Note the first if in Matches.

Comment 20 by Paul.A.Lalonde:

a)  Yes, I think that's the required behavior.
b) I think the right answer is that if *any* of the IP addresses is in the no_proxy
block we should connect on that address and not proxy it.  I think anything else would
keep me from definitively pointing at a site without a proxy.
c) I this my response at b) requires caching, or a different IP address might be chosen.

Comment 21 by Paul.A.Lalonde:

Russ - That breaks my use case: I want anything on the inside of my firewall to be
no_proxy, no matter how the address is specified.  That differs from chrome's behavior,
apparently.  
I no longer work behind that firewall, or on the code base, so it's a little hard for me
to replicate my old use case :-(

Comment 22:

Paul, this bug is about implementing the no_proxy environment variable's behavior
consistently with regard to other popular HTTP clients.
Which HTTP client(s) did you use previously which did DNS lookups on hostnames before
matching no_proxy?

Comment 23 by Paul.A.Lalonde:

Brad - I thought it was Chrome. I was keeping 10.0.0.0/8 in my no_proxy to avoid hitting
a proxy on any .intel.com sites; I don't know which code path was being used to make
that happen.  I can't easily replicate the experiment now that I'm no longer at Intel
(MS seems to be a bit smarter/more transparent on its proxy redirection goop).

Comment 24:

Python:
def proxy_bypass_environment(host):
    """Test if proxies should not be used for a particular host.
    Checks the environment for a variable named no_proxy, which should
    be a list of DNS suffixes separated by commas, or '*' for all hosts.
    """
    no_proxy = os.environ.get('no_proxy', '') or os.environ.get('NO_PROXY', '')
    # '*' is special case for always bypass
    if no_proxy == '*':
        return 1
    # strip port off host
    hostonly, port = splitport(host)
    # check if the host ends with any of the DNS suffixes
    for name in no_proxy.split(','):
        if name and (hostonly.endswith(name) or host.endswith(name)):
            return 1
    # otherwise, don't bypass
    return 0
If the environment is not set then it falls back to system-specific things on Windows
and Mac. But the environment overrides even those. There is not even CIDR code.

Comment 25:

Further problems with $no_proxy, not sure if I should file a separate bug for this or if
it can be addressed here. It seems currently that Go's code checks for domains with a
leading dot. So for example if no_proxy=example.com then a request for foo.example.com
still goes through the proxy. However if no_proxy=.example.com then it doesn't go
through the proxy. I don't think other clients have the same requirement.

Comment 26:

Kamil, open a new issue, but please include in your report the actual behavior of at
least one (ideally more) other HTTP client.

Comment 27:

Brad: filed https://golang.org/issue/4574

Comment 28:

Closing this, as it appears nobody else does DNS-before-matching.

Status changed to Invalid.