If it is a regular expression, then it is not able to match all valid email addresses, because the grammar of email addresses is context-free, and regular expressions can only match regular grammars. It doesn't matter if it is defined or not: if it's a true regular expression, then it simply cannot validate email addresses.
(it may, of course, be a context-free expression masquerading as a regular expression)
I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)
I suspect that it will, but of course I could be wrong.
I don't think you're really correct about "email addresses" being context-free, or at least, citation, please?
When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`. I don't think most people are going to think they can enter that, nor would most programmers even think to implement it. And I certainly wouldn't want to try explaining it to a PM…
If you accept that assumption, what about `addr-spec` isn't regular?
Also, using that assumption, your "perfectly valud email addresses such as …" would appear to not be valid, as it has unbalanced quotes. (In fact, even under the grammar of `address`, I'm not sure it's valid; it feels like it should be invalid for the same reason, but I've not rigorously checked this.)
> When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`.
Well, sure. Let's look at what RFC 5322 defines as an addr-spec[1]:
addr-spec = local-part "@" domain
And how does it define a local-part? local-part = dot-atom / quoted-string / obs-local-part
Let's ignore quoted-string and obs-local-part for the moment. What is a dot-atom? dot-atom = [CFWS] dot-atom-text [CFWS]
And what is CFWS? CFWS = (1*([FWS] comment) [FWS]) / FWS
What's a comment? comment = "(" *([FWS] ccontent) [FWS] ")"
So far, all of this has been matchable with a regular expression. But what's a ccontent? ccontent = ctext / quoted-pair / comment
See that there? A comment is composed of a balanced pair of parentheses around, perhaps, another comment! Thus (this (is (a (heavily (commented (email \(address))))))[email protected](some more (to prove (the point))) is a perfectly viable RFC5322 address!Pair-balancing, of course, is impossible with regular expressions, since matching pairs requires push-down automata (which match CFGs) and cannot be done with finite-state machines, (which match regular expressions).
QED.
> Also, using that assumption, your "perfectly valud[sic] email addresses such as …" would appear to not be valid, as it has unbalanced quotes.
Nope, there are no unbalanced quotes in (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest): the first quote balances with the third, while the second quote is one of a quoted pair \" (which is allowed within a quoted-string, which is allowed within a local-part). It's all allowed per the spec.
I'll admit that it's a bit surprising, but it's true. One simply cannot match a valid RFC5322 addr-spec with a regular expression. One can, of course, match it with something which pretends to be regular but isn't really (as I noted).
Well done.
You can find multiple short email validation snippets using Lua LPeg pretty easily, but this is from the Lua creator's talk about LPeg which includes a part about proper RFC822 validation and how complicated it is for regex, but can be concisely done with PEGs.
http://program-transformation.org/pub/WGLD/Austin2012/Robert...
Here is the entire PEG slide. (Looks CFG-ish to me.)
address <- mailbox / group
group <- phrase ":" mailboxes? ";"
phrase <- word ("," word?)*
mailboxes <- mailbox ("," mailbox?)*
mailbox <- addr_spec / phrase route_addr
route_addr <- "<" route? addr_spec ">"
route <- ("@" domain) ("," ("@" domain)?)* ":"
addr_spec <- local_part "@" domain
local_part <- word ("." word)*
domain <- sub_domain ("." sub_domain)*
sub_domain <- domain_ref / domain_literal
domain_ref <- atom
domain_literal <- "[" ([^][] / "\" .)* "]"
word <- atom / quoted_string
atom <- [^] %c()<>@,;:\".[]+
quoted_string <- '"' ([^"\%nl] / "\" .)* '"'Are those parens really permitted in the host name? I'm looking at RFC 3696.
> 3. Restrictions on email addresses
> .... The syntax of the domain part corresponds to that in the previous section.
and
> 2. Restrictions on domain (DNS) names
> Any characters, or combination of bits (as octets), are permitted in DNS names. However, there is a preferred form that is required by most applications. This preferred form has been the only one permitted in the names of top-level domains, or TLDs. .... The LDH rule, as updated, provides that the labels (words or strings separated by periods) that make up a domain name must consist of only the ASCII [ASCII] alphabetic and numeric characters, plus the hyphen. No other symbols or punctuation characters are permitted, nor is blank space.
> Are those parens really permitted in the host name? I'm looking at RFC 3696.
RFC5322 states that an addr-spec consists of a local-part, followed by @, followed by a domain. It states that a domain may be a domain-literal; a domain-literal may begin and end with commented folding whitespace (CFWS). It's all in https://tools.ietf.org/html/rfc5322#section-3.4.1
It's only the dtext portion of the domain-literal which must consist of printable ASCII characters, not including [, ] or \.
Yes, foo@bar!baz!quux is a valid RFC5322 email address (decimal 33, i.e. !) is allowed, per the dtext production of the spec) — and so is foo@(comment (nested comment (escaped \(comment)))bar!baz!quux(another (comment (to prove) the point))
Whether one should ever actually use such an address is, of course, another matter entirely.
The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting.
It is! Conceivably, it could be used to implement a new kind of domain-less mail system, e.g. foo@$megamail or something.
If it doesn't, remind me to patch it to not accept that.
I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.
You'd be wrong to do so. The whole point of RFCs and Standards is to take things out of the realm of personal preference.
Also, I suspect that a validator which allowed reasonable addresses like jim(somesite)@foo.invalid (which is both a good use of comments and what the + hack emulates) or "Ted Smith"@bar.invalid or "work@home"@jobs.invalid or "William \"Bill\" Jones"@baz.invalid but disallowed unreasonable ones would also be able to solve the halting problem. How many perfectly reasonable allowed features must one use before it's considered abuse?
> I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.
An email address is, simply, an address which can be given to an Internet mail server in order for it to properly route an email. All of the addresses I gave are perfectly valid ways to do so. An addresses is a property of the addressee, not of the addressor.
In other words: it may be your site, but it's my address.
Many of the more arcane things that can technically be done in an email address seem to me to be in the "turned out to be bad ideas" bin, and I have no problem making them be de facto deprecated even if no RFC has yet caught up to that.
Also, no halting-problem issues at all; my preferred approach would be to disallow certain classes of characters.
That may be true, but it's something else entirely for someone to drop them because the problems they cause … were enabled by him. An email address using characters you don't like is perfectly valid, and perfectly deliverable, right up until you refuse to deliver it. It doesn't cause any problems until you decide to make it cause a problem.
> Many of the more arcane things that can technically be done in an email address seem to me to be in the "turned out to be bad ideas" bin, and I have no problem making them be de facto deprecated even if no RFC has yet caught up to that.
> Also, no halting-problem issues at all; my preferred approach would be to disallow certain class
I really don't see why you think it's a bad idea to disallow more than alphanumeric (and plus? and ampersand?) characters in the local-part. Cui malo?
I prefer this because I work in a world where email addresses are provided to me, and possibly stored and then retrieved and used and even displayed, as text, and the RFCs allow some nightmarish things when you consider the interaction of the syntax the RFCs permit and the set of characters which are sensitive to one or more of the non-MTA components of that chain.
Your original example, for instance, contains characters that require escaping or at least careful handling for multiple situations, and even goes so far as to contain things that will be interpreted as escape sequences in some contexts. I'm sure that Robert(';DROP TABLE users;)(<script type="text/javascript">alert("Bobby Tables")</script>)@not.a.hacker(honest) will be terribly disappointed to know that he needs to use a different address to register on my site. I'll send him over to you instead.
Technically-correct is the best kind of correct:-)
But I do think it's important to note, since there really are differences between regular and 'regular' expressions.
I wish that were the case. http://www.regular-expressions.info/refunicode.html
(They used to have a much more useful and concise comparison table but I can't for the life of me find it.)
I guess what I really meant is that programmers are talking about PCRE in terms of power, not in terms of exact syntax. In particular, they have recursive patterns, which are sufficient to pull them up to context-free grammars.
>>> from django.core.validators import EmailValidator
>>> EmailValidator()(""""()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)""")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/django/core/validators.py", line 203, in __call__
raise ValidationError(self.message, code=self.code)
django.core.exceptions.ValidationErrorSeriously, though, this is why regexp-based validation is Not Even Wrong. It's like showing up to a gunfight with a keen eye for detail.
2. Remove everything between + and @
3. Do validation
I'd never looked closely at Django's email validation, but after looking at it I'm inclined to stop using it. Example reason:
> # max length for domain name labels is 63 characters per RFC 1034
There is a high likelihood that most of your users have shorter domain names, and also a high likelihood that most if not all registrars currently enforce this. But it also used to be highly likely that domains didn't begin with a numeral, and even though it was disallowed, 3m.com existed.
Another: neither the tests nor the patterns account for unicode, which is valid for domain names irrespective of the local part, and if not allowed for the local part by existing providers almost certainly will be eventually.
Also, a thing I noticed in the Django tests that is just upsetting:
- name: `TEST_DATA`
- values: `(validator, value, expected)` (a validator of the data is not the data)
- unstructured groups of such that should almost certainly be structured
Without protocol changes, DNS cannot support labels longer than 63 bytes.
However, I agree that validating it is unnecessary.
Back in the 90s, we ran the customer rewards program mailing list for a mainstream business you've heard of. A [gnarly] regex took care of the gross failures but we still had double-digit percentage of invalid addresses and many spam reports because people mistyped their username, used their old ISP's email address which had been disconnected, etc.
The only approach which produced satisfactory results back then was to have a bit of (horrible) PHP code which did a full SMTP connection to deliver the welcome email. Even that wasn't enough to ensure you'd delivered it to the right person, however, so we had to track clicks on an activation link.
I would be surprised if the average internet user has gotten significantly more reliable over the last couple of decades and that's born out by the wide variety of misdirected legitimate email my [email protected] account receives.
The other key step is having some contact form which doesn't require the same test so people can report bugs if you get the logic wrong.
2. Getting billed for sending emails? WTF?
Email validation is actually really important if you send even a medium amount of email. Testing by bouncing is not a very good idea, especially since there are a lot of ways to test the deliverability of the email before you try and send it.
That's not to say use a regex instead, you can do some simple checks on the domain etc or you can use one of the many services to check for you (for of a fee of course).
It's understandable not to know this unless you've run a medium - large site, so here's some best practices to follow: https://documentation.mailgun.com/best_practices.html#email-...
?!?
OK, let's break this down: What could be reasons for an email to be bounced due to an invalid address?
1. Because the address is syntactically invalid: Your own email server will notice and reject/bounce the email. Noone outside your own server will notice.
2. Because the domain doesn't exist or has no inbound mailserver: Your own email server will do the DNS lookup(s) and bounce the email. Noone outside your own server will make any note of that (and you can't find out without doing DNS lookups anyway).
3. Because the localpart doesn't exist: You can't find that out without trying to actually send an email (and consequently risking a bounce).
So, what exactly was the point of email validation "without bounces" again?
The only thing you can actually do to avoid unnecessary bounces is to make sure your emails have a valid return path and to make sure that you somehow process all the bounces that arrive there to make sure that you don't continue sending emails to addresses that start sending bounces (i.e., that don't exist (any more)). If you send lots of mails, it's probably best to employ VERP for that.
An email address which doesn't validate against the applicable RFC is ipso facto obviously invalid.
Validating against the RFC is somewhat complex though.
Hence, not obvious :-)
If there is one thing we can learn from all the misguided attempts at validating email addresses that we have seen over the years, it is that it is obviously not obvious which email addresses are syntactically invalid.
That said, "foot@bar." may be valid, but ICANN says it should not be functional. They prohibit dotless domain names (that is, top-level domain names may not have A, AAA, or MX records).
"foo", ok, I agree with the blogpost, if there isn't even an @ in there somewhere, maybe it's sensible to catch that and reject it outright. But that's probably about it.
Also whether you use RegExp on email or not totally depends on what you're doing with the address. Are you throwing that email into Salesforce? Great! Salesforce does RegExp validation on all email addresses and the field is required. You _HAVE_ to do it. Solution? Use the same validation that Salesforce does. That's what I do.
Also the OPs math about how common invalid email addresses are is wrong. Do you know what the most common way of screwing up an email address is? Hitting tilde when you're tabbing away from the field. And that error is in the tenths or hundreds of a percent range (which is still significant). I have a dataset of 5MM email addresses typed in by jaded call center agents to prove it.
Yes, they aren't special, and you shouldn't validate any of the other stuff either, unless you actually need to understand the semantics. So much breakage happens because people think that they need to validate all kinds of stuff, it's not funny. People have their legal real names rejected because they're "not a valid name", people have their correct postal address rejected because it's "not a valid postal address" ... just forget it, if someone tells you their name, just believe them it's their name, if someone tells you their postal address, just believe them it's their postal address, if someone tells you their email address ...
Oh, and don't ever think about "sanitizing" stuff. Just don't. If you think you need it, you are doing something wrong. The solution to SQL injection is not to disallow people using "'" characters in messages, the solution to cross site scripting is not to prevent people from using "<" characters in comments, ... that's what encoding/escaping is for.
I ran a fresh install of Django to confirm and it behaves as you suggest.
Edit: Perhaps only for low digits. Hmm.
>>> from django.core.validators import EmailValidator
>>> e = EmailValidator()
>>> e.__call__('foo@[1.2.3.4]')
>>> e.__call__('[email protected]')
ValidationError: [u'Enter a valid email address.']
>>> e.__call__('[email protected]')
ValidationError: [u'Enter a valid email address.']
>>> e.__call__('foo@[0.0.0.0]')
>>> e.__call__('[email protected]')
>>> e.__call__('foo@[10.10.10.10]')
>>> e.__call__('[email protected]')
>>> e.__call__('foo@[255.255.255.255]')
Email addresses aren't a special enough case to be handled differently than any other user input, which we always validate to both sanitize and show client-side errors, if nothing else.
Sure, the complete regex is complex, but it is defined and is hardly unconquerable. Look at Django's `EmailValidator` implementation for example [0] that is mature and well tested [1].
The author has not convinced me that ignoring validation is the right choice when options with a scope so thorough exist.
[0]: https://github.com/django/django/blob/master/django/core/val...
[1]: https://github.com/django/django/blob/a9215b7c36bff232bcc941...