net/smtp: add Hello

net/smtp: add Hello

[dho, 2012-01-18 14:50:42]

LGTM otherwise

http://codereview.appspot.com/5555045/diff/4001/src/pkg/net/smtp/smtp.go
File src/pkg/net/smtp/smtp.go (right):

http://codereview.appspot.com/5555045/diff/4001/src/pkg/net/smtp/smtp.go#newc...
src/pkg/net/smtp/smtp.go:83: // any of the other methods.
I think this comment is misleading. The SMTP spec does tell us what is valid
(and required) to pass here, and if we know this information, is there a
compelling argument to not provide it by default?

As mentioned in the email thread spawning this discussion, one must provide the
name corresponding to the reverse record of the client's connecting IP, or the
server's hostname, or the SMTP-valid IP literal, in that order of preference.

At best, it's not right to say this is "only necessary if the client needs
control over the host name used" because the SMTP specification specifies what
someone must send here, and in a production environment, that is *never* going
to be "localhost." We should suggest calling this method to set the hostname as
per the ordered list I specified above.

[rsc, 2012-01-18 14:54:51]

For what it's worth, I disagree about leaking os.Hostname.
If the caller wants to do that explicitly, fine.  But I don't
think we should do it without being told to.  SMTP dates
from a much more trusting time.

I am happy to send "[127.0.0.1]" instead of "localhost".
RFC 5321 definitely allows that.

Russ

[rsc, 2012-01-18 15:22:59]

*** Abandoned ***

[dho, 2012-01-18 17:19:31]

2012/1/18 Russ Cox <rsc@golang.org>:
> For what it's worth, I disagree about leaking os.Hostname.
> If the caller wants to do that explicitly, fine.  But I don't
> think we should do it without being told to.  SMTP dates
> from a much more trusting time.
>
> I am happy to send "[127.0.0.1]" instead of "localhost".
> RFC 5321 definitely allows that.

It does, but it may not be correct. I don't understand your
disagreement either, this is specifically the order that RFC 5321
specifies, which upon re-reading is ambiguous at best with regards to
using what amounts to os.Hostname in EHLO (in the section 2.3.5 I cite
below), so I guess I withdraw that part.

Section 2.3.5

  o  The domain name given in the EHLO command MUST be either a primary
      host name (a domain name that resolves to an address RR) or, if
      the host has no name, an address literal, as described in
      Section 4.1.3 and discussed further in the EHLO discussion of
      Section 4.1.4.

Section 4.1.4

   The SMTP client MUST, if possible, ensure that the domain parameter
   to the EHLO command is a primary host name as specified for this
   command in Section 2.3.5.  If this is not possible (e.g., when the
   client's address is dynamically assigned and the client does not have
   an obvious name), an address literal SHOULD be substituted for the
   domain name.

In the case that we're substituting an address literal, we should
substitute a correct one. If you're connecting  on IPv6 localhost, for
instance, the literal should be [IPv6:::1]. I don't see the point in
fixing this issue if we're going to fix it by continuing to send
incorrect information by default. I have contractual obligations that
prevent me from contributing SMTP-related code, otherwise I would
submit another CL.

> Russ

--dho

[rsc, 2012-01-18 17:24:41]

SMTP predates VPNs and NATs.
I do not want to send

HELO my.internal.host.name.company.com

nor do I really want to send (on a dialup)

HELO 24y0ioreqnalfgadjna.comcast.net

I want to send something that is legal according
to the spec but reveals no information about my
internal network.  Servers shouldn't care what I say
here, and for those that do, there is the new method.

I don't really care what the default is, as long as it
is a fixed string.  "localhost", "[127.0.0.1]", "[10.0.0.0]",
whatever.

Russ

[dho, 2012-01-18 17:46:07]

2012/1/18 Russ Cox <rsc@golang.org>:
> SMTP predates VPNs and NATs.

But RFC 5321 does not, it is a revision of SMTP as of 2008.

> I do not want to send
>
> HELO my.internal.host.name.company.com
>
> nor do I really want to send (on a dialup)
>
> HELO 24y0ioreqnalfgadjna.comcast.net

But this is exactly what is required and exactly what carriers want
and need people to do, and there is very good reason for this. In a
practical scenario, you are not revealing anything about your internal
topology to anyone external, and if you are, your mail setup is broken
and you *should* be dinged for spamming.

> I want to send something that is legal according
> to the spec but reveals no information about my
> internal network.  Servers shouldn't care what I say
> here, and for those that do, there is the new method.

Servers *do* care about the spec because of the problem of spam. If
you are sending mail from an internal network, you are probably
sending it *though* an MX gateway that is designed to relay your mail
to where ever it needs to go (and *that* MX may send to another MX in
between there and the destination, depending on the network topology).
Your gateway MX may care about the HELO/EHLO string to ensure that you
aren't trying to spoof connecting from inside their network.

There is no good reason that anybody using this API for legitimate
purposes should be sending directly to edge MXes, and a developer's
ignorance of how SMTP works and how the current email environment
functions isn't a good reason for us to make an API that violates what
the spec says for their convenience. An API should make the
developer's life easier, not harder. Requiring the developer to know
whether or not they are trying to send production mail and maintain a
good reputation effectively makes it more difficult for a developer to
write SMTP applications with Go. Providing an API that conforms to the
spec encourages senders to set up a network that allows them to
effectively and easily send mail without worrying about having their
sending IPs put on an RBL for exhibiting spoofing behavior.

If a user is concerned about revealing information about their
internal network topology, they should learn more about SMTP and
figure out how to build an infrastructure that embraces how mailers
and carriers actually use mail instead of try to code around it.

> I don't really care what the default is, as long as it
> is a fixed string.  "localhost", "[127.0.0.1]", "[10.0.0.0]",
> whatever.

I stand by my assertion that the default should conform to the spec
for the above reasons.

> Russ

--dho

[rsc, 2012-01-18 17:48:47]

Let's suppose I am a Comcast customer running a NAT at home.
My computer is 192.168.0.10.  It has no idea what its "real" internet
name is.  What should it say when it talks to the Comcast MX server?
Why is that helpful?

Russ

[dho, 2012-01-18 19:02:52]

2012/1/18 Russ Cox <rsc@golang.org>:
> Let's suppose I am a Comcast customer running a NAT at home.
> My computer is 192.168.0.10.  It has no idea what its "real" internet
> name is.  What should it say when it talks to the Comcast MX server?
> Why is that helpful?

Per the spec, it should say either a reverse record that it knows
about itself or [192.168.0.10].

This is not the general case of who we should expect to be sending
mail. Are we really expecting to provide an API for a
high-performance, concurrent language to cater to people trying to
send email from residential connections? We should release an API that
caters to people doing production work. If someone doesn't want to
leak information about their residential topology to a provider that
gives them no SLA, then I would argue that it is *that* person who
should be overriding the default value with something not conforming
to the spec.

I really don't understand the angle of this argument for a lot of
reasons. It's not that it's helpful for the cases you are mentioning.
It largely doesn't make a difference for cases like this, because most
ISPs aren't going to let you connect to someone else's SMTP server
anyway as an anti-spam measure. My argument is that this isn't the
standard use case for such an API and that what I am suggesting is
helpful for the cases that you aren't mentioning, which easily account
for over 99.999% of legitimate email that goes over the Internet. Most
people are going to use an API (and especially the one provided with a
language like Go) to send mail over production networks, and not
sourcing from residential lines.

> Russ

--dho

[rsc, 2012-01-18 19:11:36]

I don't believe it is net/smtp's job to leak information
about the machine it is running on, unless asked to.
It's that simple.

It's silly for servers to care what the client says here.
The server can just as easily look at the connecting
IP address and deduce whatever it likes.

Russ

[dho, 2012-01-18 19:27:51]

2012/1/18 Russ Cox <rsc@golang.org>:
> I don't believe it is net/smtp's job to leak information
> about the machine it is running on, unless asked to.
> It's that simple.

*sigh* In 99.9999% of cases it isn't doing this, and by violating the
spec, you are screwing over legitimate senders who are operating on
production networks, assuming that the library conforms to the spec.
The library says it conforms to the spec in documentation ("Package
smtp implements the Simple Mail Transfer Protocol as defined in RFC
5321"). It doesn't do this, and if the decision is to not conform to
the spec, this needs to be removed and explain how it deviates so that
people can write their applications accordingly. It isn't the job of
API implementers to decide which parts of the spec to adhere to, nor
should it be their decision to make an API easier to use in a
fundamentally broken environment.

> It's silly for servers to care what the client says here.
> The server can just as easily look at the connecting
> IP address and deduce whatever it likes.

You've just identified how this is used to combat spammers: this
string is used in policies, being compared to the connecting address,
and if they don't match, it dings a spam score or issues a 421
disconnect. I can't write code for this because it falls within the
realm of my employer's objectives (I might be able to get an exemption
here, but it would take time), but as someone who works on email
products I would really advise not breaking the specification. It
makes things harder for those of us who write mail servers and for
people who administer them. All you do by providing a bogus default
value is discourage people from setting up proper MX environments.

Please don't break adherence to the spec. I'm sorry if this is coming
across as inflammatory; I obviously have an interest in email and its
proper function, and being non-conforming in this fashion is enough to
discourage people from using Go to write applications that need to
speak SMTP.

--dho

> Russ

[rsc, 2012-01-18 19:32:21]

Thanks.  I will try to gather some more opinions.

[dho, 2012-01-18 20:03:03]

2012/1/18 Russ Cox <rsc@golang.org>:
> Thanks.  I will try to gather some more opinions.

Thanks for the consideration. FWIW, I looked into some other
implementations, and I'd argue that some of them are broken, but
here's the findings:

 * Python does what I consider to be the right thing.
 * PHP calls the system's mailer, but (surprisingly) does what I
consider to be the right thing if you configure it to speak directly
to an MX.
 * Ruby defaults to localhost but documents that you should normally
set this string to the proper value.
 * Luasocket tries to do it right right, either using SERVER_NAME from
the environment (assuming you're running as a CGI) or falling back to
localhost.
 * Perl's Net::SMTP defaults to localhost.localdomain and does not
document how this deviates from the spec. I think this is horribly
broken, obviously.

So I guess API implementers are about half split on this. At the very
least, it should be documented that you should set this value to the
connecting IP's DNS RR, or the ip literal (with examples for v4 and
v6) for normal production traffic.

--dho

[rsc, 2012-01-19 21:59:53]

I talked about this with various people with strong opinions
here, and it was about evenly split.  I am inclined to leave it
as "localhost" and document that if you're talking to general
MX servers you will want to override this.  If you're doing
something like authenticated SMTP to a local system, then
it doesn't matter as much.

Russ

[rsc, 2012-01-19 22:00:04]

Would you like to pick up this CL and write the Hello comment?