News for hackers. Stuff that matters.
Re:The Correct Way to Validate Email Addresses by wtbob on Wednesday September 07, 2016 @08:32PM (#12449262)
> Sure, the complete regex is complex, but it is defined and is hardly unconquerable.

If it is a regular expression, then it is not able to match all valid email addresses, because the grammar of email addresses is context-free, and regular expressions can only match regular grammars. It doesn't matter if it is defined or not: if it's a true regular expression, then it simply cannot validate email addresses.

(it may, of course, be a context-free expression masquerading as a regular expression)

I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)

I suspect that it will, but of course I could be wrong.

Re:The Correct Way to Validate Email Addresses by deathanatos on Wednesday September 07, 2016 @09:22PM (#12449493)
> the grammar of email addresses is context-free

I don't think you're really correct about "email addresses" being context-free, or at least, citation, please?

When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`. I don't think most people are going to think they can enter that, nor would most programmers even think to implement it. And I certainly wouldn't want to try explaining it to a PM…

If you accept that assumption, what about `addr-spec` isn't regular?

Also, using that assumption, your "perfectly valud email addresses such as …" would appear to not be valid, as it has unbalanced quotes. (In fact, even under the grammar of `address`, I'm not sure it's valid; it feels like it should be invalid for the same reason, but I've not rigorously checked this.)

[1]: https://tools.ietf.org/html/rfc5322#section-3.4.1

Re:The Correct Way to Validate Email Addresses by wtbob on Wednesday September 07, 2016 @10:48PM (#12449903)
> I don't think you're really correct about "email addresses" being context-free, or at least, citation, please?

> When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`.

Well, sure. Let's look at what RFC 5322 defines as an addr-spec[1]:

    addr-spec       =   local-part "@" domain
And how does it define a local-part?

    local-part      =   dot-atom / quoted-string / obs-local-part
Let's ignore quoted-string and obs-local-part for the moment. What is a dot-atom?

    dot-atom        =   [CFWS] dot-atom-text [CFWS]
And what is CFWS?

    CFWS            =   (1*([FWS] comment) [FWS]) / FWS
What's a comment?

    comment         =   "(" *([FWS] ccontent) [FWS] ")"
So far, all of this has been matchable with a regular expression. But what's a ccontent?

    ccontent        =   ctext / quoted-pair / comment
See that there? A comment is composed of a balanced pair of parentheses around, perhaps, another comment! Thus (this (is (a (heavily (commented (email \(address))))))[email protected](some more (to prove (the point))) is a perfectly viable RFC5322 address!

Pair-balancing, of course, is impossible with regular expressions, since matching pairs requires push-down automata (which match CFGs) and cannot be done with finite-state machines, (which match regular expressions).

QED.

> Also, using that assumption, your "perfectly valud[sic] email addresses such as …" would appear to not be valid, as it has unbalanced quotes.

Nope, there are no unbalanced quotes in (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest): the first quote balances with the third, while the second quote is one of a quoted pair \" (which is allowed within a quoted-string, which is allowed within a local-part). It's all allowed per the spec.

I'll admit that it's a bit surprising, but it's true. One simply cannot match a valid RFC5322 addr-spec with a regular expression. One can, of course, match it with something which pretends to be regular but isn't really (as I noted).

Re:The Correct Way to Validate Email Addresses by deathanatos on Thursday September 08, 2016 @01:46AM (#12450564)
Oh, shoot, I missed that. I saw "FWS" (meaning "folding white space"), and assumed that didn't include comments since they have nothing to do with folding whitespace.

Well done.

Re:The Correct Way to Validate Email Addresses by wtbob on Thursday September 08, 2016 @06:54AM (#12451828)
No worries. The RFC is … complex.
Re:The Correct Way to Validate Email Addresses by dottrap on Wednesday September 07, 2016 @10:34PM (#12449831)
I'm not sure if they are context-free, but talks about Parseable Expression Grammars, specifically Lua's LPeg, implies they might be. Conceptually, PEGs are to Context Free Grammars as RegEx is to Regular Expressions.

You can find multiple short email validation snippets using Lua LPeg pretty easily, but this is from the Lua creator's talk about LPeg which includes a part about proper RFC822 validation and how complicated it is for regex, but can be concisely done with PEGs.

http://program-transformation.org/pub/WGLD/Austin2012/Robert...

Here is the entire PEG slide. (Looks CFG-ish to me.)

    address <- mailbox / group
    group <- phrase ":" mailboxes? ";"
    phrase <- word ("," word?)*
    mailboxes <- mailbox ("," mailbox?)*
    mailbox <- addr_spec / phrase route_addr
    route_addr <- "<" route? addr_spec ">"
    route <- ("@" domain) ("," ("@" domain)?)* ":"
    addr_spec <- local_part "@" domain
    local_part <- word ("." word)*
    domain <- sub_domain ("." sub_domain)*
    sub_domain <- domain_ref / domain_literal
    domain_ref <- atom
    domain_literal <- "[" ([^][] / "\" .)* "]"
    word <- atom / quoted_string
    atom <- [^] %c()<>@,;:\".[]+
    quoted_string <- '"' ([^"\%nl] / "\" .)* '"'
Re:The Correct Way to Validate Email Addresses by tedmiston on Wednesday September 07, 2016 @09:20PM (#12449483)
> ...@(valid)example.org(honest)

Are those parens really permitted in the host name? I'm looking at RFC 3696.

> 3. Restrictions on email addresses

> .... The syntax of the domain part corresponds to that in the previous section.

and

> 2. Restrictions on domain (DNS) names

> Any characters, or combination of bits (as octets), are permitted in DNS names. However, there is a preferred form that is required by most applications. This preferred form has been the only one permitted in the names of top-level domains, or TLDs. .... The LDH rule, as updated, provides that the labels (words or strings separated by periods) that make up a domain name must consist of only the ASCII [ASCII] alphabetic and numeric characters, plus the hyphen. No other symbols or punctuation characters are permitted, nor is blank space.

https://tools.ietf.org/html/rfc3696#section-2

Re:The Correct Way to Validate Email Addresses by wtbob on Wednesday September 07, 2016 @10:59PM (#12449959)
> > ...@(valid)example.org(honest)

> Are those parens really permitted in the host name? I'm looking at RFC 3696.

RFC5322 states that an addr-spec consists of a local-part, followed by @, followed by a domain. It states that a domain may be a domain-literal; a domain-literal may begin and end with commented folding whitespace (CFWS). It's all in https://tools.ietf.org/html/rfc5322#section-3.4.1

It's only the dtext portion of the domain-literal which must consist of printable ASCII characters, not including [, ] or \.

Yes, foo@bar!baz!quux is a valid RFC5322 email address (decimal 33, i.e. !) is allowed, per the dtext production of the spec) — and so is foo@(comment (nested comment (escaped \(comment)))bar!baz!quux(another (comment (to prove) the point))

Whether one should ever actually use such an address is, of course, another matter entirely.

Re:The Correct Way to Validate Email Addresses by tedmiston on Thursday September 08, 2016 @02:48PM (#12456028)
Thank you for that; this has been eyeopening.

The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting.

Re:The Correct Way to Validate Email Addresses by wtbob on Thursday September 08, 2016 @08:12PM (#12458672)
> The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting.

It is! Conceivably, it could be used to implement a new kind of domain-less mail system, e.g. foo@$megamail or something.

Re:The Correct Way to Validate Email Addresses by ubernostrum on Thursday September 08, 2016 @01:25AM (#12450485)
I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)

If it doesn't, remind me to patch it to not accept that.

I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.

Re:The Correct Way to Validate Email Addresses by wtbob on Thursday September 08, 2016 @07:01AM (#12451866)
> If it doesn't, remind me to patch it to not accept that.

You'd be wrong to do so. The whole point of RFCs and Standards is to take things out of the realm of personal preference.

Also, I suspect that a validator which allowed reasonable addresses like jim(somesite)@foo.invalid (which is both a good use of comments and what the + hack emulates) or "Ted Smith"@bar.invalid or "work@home"@jobs.invalid or "William \"Bill\" Jones"@baz.invalid but disallowed unreasonable ones would also be able to solve the halting problem. How many perfectly reasonable allowed features must one use before it's considered abuse?

> I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.

An email address is, simply, an address which can be given to an Internet mail server in order for it to properly route an email. All of the addresses I gave are perfectly valid ways to do so. An addresses is a property of the addressee, not of the addressor.

In other words: it may be your site, but it's my address.

Re:The Correct Way to Validate Email Addresses by ubernostrum on Thursday September 08, 2016 @01:18PM (#12454987)
There are already things that are technically allowed by RFCs for various types of input but should be disallowed because they cause problems. So something being in an RFC doesn't create any kind of binding requirement, and very often the things that get dropped are things that, on contact with the real world, turned out to be bad ideas.

Many of the more arcane things that can technically be done in an email address seem to me to be in the "turned out to be bad ideas" bin, and I have no problem making them be de facto deprecated even if no RFC has yet caught up to that.

Also, no halting-problem issues at all; my preferred approach would be to disallow certain classes of characters.

Re:The Correct Way to Validate Email Addresses by wtbob on Thursday September 08, 2016 @08:19PM (#12458708)
> There are already things that are technically allowed by RFCs for various types of input but should be disallowed because they cause problems.

That may be true, but it's something else entirely for someone to drop them because the problems they cause … were enabled by him. An email address using characters you don't like is perfectly valid, and perfectly deliverable, right up until you refuse to deliver it. It doesn't cause any problems until you decide to make it cause a problem.

> Many of the more arcane things that can technically be done in an email address seem to me to be in the "turned out to be bad ideas" bin, and I have no problem making them be de facto deprecated even if no RFC has yet caught up to that.

> Also, no halting-problem issues at all; my preferred approach would be to disallow certain class

I really don't see why you think it's a bad idea to disallow more than alphanumeric (and plus? and ampersand?) characters in the local-part. Cui malo?

Re:The Correct Way to Validate Email Addresses by ubernostrum on Saturday September 10, 2016 @12:37AM (#12467706)
I would prefer to allow plus, dot, hyphen, underscore, and alphanumeric characters in the local part and probably not much of anything else.

I prefer this because I work in a world where email addresses are provided to me, and possibly stored and then retrieved and used and even displayed, as text, and the RFCs allow some nightmarish things when you consider the interaction of the syntax the RFCs permit and the set of characters which are sensitive to one or more of the non-MTA components of that chain.

Your original example, for instance, contains characters that require escaping or at least careful handling for multiple situations, and even goes so far as to contain things that will be interpreted as escape sequences in some contexts. I'm sure that Robert(';DROP TABLE users;)(<script type="text/javascript">alert("Bobby Tables")</script>)@not.a.hacker(honest) will be terribly disappointed to know that he needs to use a different address to register on my site. I'll send him over to you instead.

Re:The Correct Way to Validate Email Addresses by thedufer on Thursday September 08, 2016 @12:15AM (#12450285)
It is true that regular expressions in the CS sense can't parse context-free grammars. However, PCRE, which is what most programmers are talking about when they say "regex", can do so. So you're both kinda right, I guess. But you're being a bit pedantic.
Re:The Correct Way to Validate Email Addresses by wtbob on Thursday September 08, 2016 @09:49AM (#12452954)
> But you're being a bit pedantic.

Technically-correct is the best kind of correct:-)

But I do think it's important to note, since there really are differences between regular and 'regular' expressions.

Re:The Correct Way to Validate Email Addresses by eyelidlessness on Thursday September 08, 2016 @12:38AM (#12450336)
> PCRE, which is what most programmers are talking about

I wish that were the case. http://www.regular-expressions.info/refunicode.html

(They used to have a much more useful and concise comparison table but I can't for the life of me find it.)

Re:The Correct Way to Validate Email Addresses by thedufer on Thursday September 08, 2016 @08:54PM (#12458893)
Interesting; I've never had to think about how regex interact with Unicode.

I guess what I really meant is that programmers are talking about PCRE in terms of power, not in terms of exact syntax. In particular, they have recursive patterns, which are sufficient to pull them up to context-free grammars.

Re:The Correct Way to Validate Email Addresses by tedmiston on Thursday September 08, 2016 @09:13PM (#12458983)
It definitely can get pedantic in a topic like this, but when all of the interesting bits are in the nuances, I don't mind so much.
Re:The Correct Way to Validate Email Addresses by Programmatic on Wednesday September 07, 2016 @08:41PM (#12449315)
You are correct:

  >>> from django.core.validators import EmailValidator
  >>> EmailValidator()(""""()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)""")
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/local/lib/python2.7/dist-packages/django/core/validators.py", line 203, in __call__
      raise ValidationError(self.message, code=self.code)
  django.core.exceptions.ValidationError
Re:The Correct Way to Validate Email Addresses by wtbob on Wednesday September 07, 2016 @09:05PM (#12449429)
Well shoot, that's my email address! /s

Seriously, though, this is why regexp-based validation is Not Even Wrong. It's like showing up to a gunfight with a keen eye for detail.

Re:The Correct Way to Validate Email Addresses by Senji on Friday September 09, 2016 @02:08AM (#12459941)
1. Strip non alphanumerics except +

2. Remove everything between + and @

3. Do validation