News for hackers. Stuff that matters.
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by deathanatos on Monday December 16, 2019 @08:34PM (#21809630)
This blog sets opacity: 0 (fully invisible) on the entire content, then fails to unset that CSS with JS, b/c the JS crashes if you block cookies.

> because the system lowercased the provided email address and compared it to the email address stored in the user database.

While sending the email to the attack-provided email, instead of the one in the database, is bad… lowercasing emails is also not valid. The lookup should never have matched in the first place.

(It's slightly more complicated: to an extent, the case of the domain name doesn't matter, ignoring non-ASCII characters — I have no idea what they do. But the local part — the portion before the @ — is case sensitive. A server is free to ignore that, and map multiple local parts to the same mailbox internally¹, and many do, but the sender cannot make that assumption.)

¹or do other weird things, like ignore dots, or +extensions, etc.

Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by jcranmer on Monday December 16, 2019 @09:10PM (#21809806)
Practically speaking, you have to treat email addresses as case-preserving: you match case-insensitive, but you always store the case the user entered in directly.

I usually give my email address as starting with a capital letter for historical reasons, and I once had an issue that wouldn't let me log on with any case whatsoever, probably because it lowercased the email address and tried to match it against the uppercase in the database.

Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by Alex3917 on Monday December 16, 2019 @09:27PM (#21809929)
> you match case-insensitive, but you always store the case the user entered in directly.

You still need to normalize the unicode for either search or display, unless you're only allowing ascii or something.

E.g. for the column you store for search and to guarantee uniqueness, casefold and then normalize to NFKC. And for the column you store for display, normalize to NFC. (And obviously you need to sanitize user input before doing anything.)

Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by marcosdumay on Monday December 16, 2019 @10:30PM (#21810255)
That's one of the reasons why it's recommended that email services to have case insensitive local parts. That and the fact that there are plenty of clients out there that will corrupt the address capitalization.
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by _jg on Monday December 16, 2019 @08:45PM (#21809695)
Unfortunately convention has normalised user expectations that email addresses are now case-insensitive. It's now a standard business requirement. Too bad few devs fully handle Unicode.
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by glass_of_water on Monday December 16, 2019 @08:57PM (#21809754)
Nit: did you mean opacity: 0?
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by deathanatos on Monday December 16, 2019 @10:16PM (#21810192)
I did, thank you.
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by iudqnolq on Monday December 16, 2019 @10:34PM (#21810279)
A few tiny nitpicks:

The local part MUST be treated as case-sensitive, servers are discouraged from doing so (as Gmail does): https://tools.ietf.org/html/rfc5321#section-2.4

Github shouldn't have normized case, but it's not insane to require lowercase in the first place I think so long as you don't convert silently.

I wouldn't call +extensions and ignoring dots weird because Gmail does that and fairly or unfairly they set the standard for user expectations of email nowadays.

Also, I think it's unreasonable to require full compliance with the spec. For example, if you go around trying to give out email addresses with comments in the ("usern(ignored_comment)[email protected]") you'll see many things break and I don't think that's an issue.

For a delightful example of how insane email addresses can get, here's a fully compliant regex to validate one:

(?:(?:\r\n)?[ \t])(?:(?:(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t] )+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))"(?:(?: \r\n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:( ?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))"(?:(?:\r\n)?[ \t])))@(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\0 31]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\ ](?:(?:\r\n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+ (?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?: (?:\r\n)?[ \t])))|(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z |(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))"(?:(?:\r\n) ?[ \t]))\<(?:(?:\r\n)?[ \t])(?:@(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\ r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n) ?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t] )))(?:,@(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t])* )(?:\.(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t] )+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t])))) :(?:(?:\r\n)?[ \t]))?(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+ |\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))"(?:(?:\r \n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?: \r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t ]))"(?:(?:\r\n)?[ \t])))@(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031 ]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\]( ?:(?:\r\n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(? :(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(? :\r\n)?[ \t])))\>(?:(?:\r\n)?[ \t]))|(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(? :(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)? [ \t]))"(?:(?:\r\n)?[ \t])):(?:(?:\r\n)?[ \t])(?:(?:(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]| \\.|(?:(?:\r\n)?[ \t]))"(?:(?:\r\n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t])(?:[^()<> @,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|" (?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))"(?:(?:\r\n)?[ \t])))@(?:(?:\r\n)?[ \t] )(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\ ".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t])(? :[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[ \]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t])))|(?:[^()<>@,;:\\".\[\] \000- \031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|( ?:(?:\r\n)?[ \t]))"(?:(?:\r\n)?[ \t]))\<(?:(?:\r\n)?[ \t])(?:@(?:[^()<>@,; :\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([ ^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\" .\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\ ]\r\\]|\\.)\](?:(?:\r\n)?[ \t])))(?:,@(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\ [\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\ r\\]|\\.)\](?:(?:\r\n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\] |\\.)\](?:(?:\r\n)?[ \t])))):(?:(?:\r\n)?[ \t]))?(?:[^()<>@,;:\\".\[\] \0 00-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\ .|(?:(?:\r\n)?[ \t]))"(?:(?:\r\n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t])(?:[^()<>@, ;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(? :[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))"(?:(?:\r\n)?[ \t])))@(?:(?:\r\n)?[ \t])* (?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\". \[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t])(?:[ ^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\] ]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t])))\>(?:(?:\r\n)?[ \t]))(?:,\s( ?:(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\ ".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))"(?:(?:\r\n)?[ \t]))(?:\.(?:( ?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[ \["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))"(?:(?:\r\n)?[ \t ])))@(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t ])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t]))(? :\.(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+| \Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t])))|(?: [^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\ ]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))"(?:(?:\r\n)?[ \t]))\<(?:(?:\r\n) ?[ \t])(?:@(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\[" ()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t]))(?:\.(?:(?:\r\n) ?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<> @,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t])))(?:,@(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@, ;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t]))(?:\.(?:(?:\r\n)?[ \t] )(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\ ".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t])))):(?:(?:\r\n)?[ \t]))? (?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\". \[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))"(?:(?:\r\n)?[ \t]))(?:\.(?:(?: \r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\[ "()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))"(?:(?:\r\n)?[ \t]) ))@(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t]) +|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t]))(?:\ .(?:(?:\r\n)?[ \t])(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z |(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)\](?:(?:\r\n)?[ \t])))\>(?:( ?:\r\n)?[ \t]))))?;\s)

http://www.ex-parrot.com/~pdw/Mail-RFC822-Address.html

Edit: Apparently Perl is wrong. Thanks https://news.ycombinator.com/item?id=21810662 !

Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by jcranmer on Monday December 16, 2019 @11:43PM (#21810662)
Except that's not actually correct, because strictly following the ABNF does not yield correct semantics for an email address.

An email address consists of a local-part, a literal @ character, and then a domain name or an IP address literal. The local-part is either a series of dot-separated atoms (/[a-zA-Z0-9!#$%&'+/=?^_`{|}~-]+/ is the syntax for an atom) or a quoted string (/"([^\\"\0-\031\x7f]|\\[^\0-\031\x7f])"/ in regex). If you support EAI, you need to add \u00a0-\u10ffff [i.e., all non-ASCII non-C1 control characters].

According to RFC 822, you can insert whitespace (and comments) arbitrarily into the mailbox production, but that is non-semantic. Seeing "From: John Doe <foo @ example (I ((hate)) CFWS).com>" means that the email address is exactly "[email protected]", and you can reject any claim of the spelling without prejudicing any email addresses.

In practice, you can drop all support for quoted strings and IP address literals in most applications. So a correct email address (pre-EAI) regex in that vein would be:

    [a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+(.[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)*@([a-zA-Z0-9]+(-[a-zA-Z0-9]+)*\.)+[a-zA-Z]{2,}
If you insist on quoted-string support, then it looks like this:

    ([a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+(.[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)*|"([^\\"\0-\031\x7f]|\\[\\"])*")@([a-zA-Z0-9]+(-[a-zA-Z0-9]+)*\.)+[a-zA-Z]{2,}
[I simplified the quoted string to only accept escapes that are semantically necessary--the strings "a b"@example.com and "a\ b"@example.com correspond to the same email address].

Edit: Sorry, there's quite a few asterisks in the regexes that Hacker News is turning into italicization, and I don't know how to unbork them.

Edit 2: Someone suggested how to unbork the standalone regexes, but the asterisks in the inline regex in the second paragraph are still missing.

Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by ComputerGuru on Tuesday December 17, 2019 @12:22AM (#21810890)
> Edit: Sorry, there's quite a few asterisks in the regexes that Hacker News is turning into italicization, and I don't know how to unbork them.

Prefix the regex with four spaces.

Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by iudqnolq on Monday December 16, 2019 @11:51PM (#21810714)
This is why I love HN. Someone replied to my joking improperly-researched comment with the right answer and a detailed explanation of why I'm wrong. I stand by my assertion that actually using that regex to check emails the user enters would be dumb.
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by chupasaurus on Tuesday December 17, 2019 @12:16AM (#21810844)
IDNs (Internationalized Domain Names) would blow any regexp to an infinity because the list of languages usable in different domains isn't unified[0]

[0]https://www.iana.org/domains/idn-tables

Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by jcranmer on Tuesday December 17, 2019 @12:34AM (#21810945)
By the time you hit IDNs, regexes for validation are no longer your biggest issue. Your real check for validity at that point becomes "can I actually contact the host" (or send an email, if validating an email address), and there is little point in aggressively validating a purported domain name instead of checking if it actually exists.
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by Avamander on Monday December 16, 2019 @11:02PM (#21810414)
Just... don't try to validate it like that. Check if you can send an e-mail to it, if you can then it's fine. I see way too many devs thinking they can validate e-mails with regex and then I can't use my own name in my e-mail.
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by iudqnolq on Monday December 16, 2019 @11:11PM (#21810476)
I agree completely. I might do .+@.+\..+ on the frontend to catch people mixing up usernames and emails, but that's it.

You might not be surprised that regex comes from a Perl module :)

Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by Avamander on Monday December 16, 2019 @11:16PM (#21810500)
To be honest, login forms should take both username and e-mail as the user identifier, far too often they don't. I personally think that the only place where such a regex should exist is during sign-up.
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by iudqnolq on Monday December 16, 2019 @11:28PM (#21810579)
Fair enough, although you could I suppose run into issues doing that retroactively if you allowed `@` in usernames...
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by fizx on Monday December 16, 2019 @11:34PM (#21810607)
That's more of a proof that regexes can't do everything, than a proof email is bad.
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by iudqnolq on Monday December 16, 2019 @11:47PM (#21810700)
Pick a language of your choice, and fully implement the spec. I bet it'll still be long.

Edit: The following is completely wrong

For example, the python module to parse an email address is around 500 lines and repeatedly warns it'll be very hard to follow without a copy of the spec in front of you. It contains code for parsing multiple timezone formats and cite to a follow up spec addressing a bug in the initial treatment of negative timezones...

https://github.com/python/cpython/blob/master/Lib/email/_par...

Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by jcranmer on Tuesday December 17, 2019 @12:20AM (#21810880)
The timezone isn't for parsing addressing headers, it's for parsing date headers. And actually, that file isn't for parsing email addresses, it's for parsing addressing headers in mail messages.

The code I wrote for parsing email headers is here: https://github.com/jcranmer/jsmime/blob/emailutils/headerpar... . A decent chunk of it is building a full lexer for email headers, and trying to cope with only supporting internationalization support in a few cases where they need to be supported. And the corner cases for that i18n support are really nasty.

Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by iudqnolq on Tuesday December 17, 2019 @12:58AM (#21811064)
Once again, you prove me completely wrong. Thanks for the detailed correction.
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by lonelappde on Monday December 16, 2019 @11:00PM (#21810404)
Please don't pollute the HN UI with garbage
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by iudqnolq on Monday December 16, 2019 @11:12PM (#21810479)
I wasn't sure about it, but even on a phone you can scroll past it in a flick and the length is a key component of why it's "interesting"
Re:Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' by paulddraper on Tuesday December 17, 2019 @02:32AM (#21811464)
Point proven I guess.