Re:Creative usernames and Spotify account hijacking
by jrochkind1 on Wednesday June 19, 2013 @08:49AM
(#5905170)
Huh, and another alternative would be using one of the unicode collation algorithms for normalization -- which, unlike UAX#15, did make it beyond draft stage to be an official part of the unicode spec.
I wonder if Unicode UTR#30 would be an alternative (more reliable?) method of idempotent normalization? http://www.unicode.org/reports/tr30/tr30-4.html
Or maybe the twisted algorithm really is UTR#30, just not labelled that?
As far as I can tell, UTR#30 did not make it to formally being part of the unicode spec, for reasons I'm not entirely clear on -- it is nonetheless quite useful, and this case is an example. Solr for instance still uses it. (http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#...).
It might be a pain to find code implementing UTR#30 in your language of choice though (I am not sure if it's part of current ICU libraries or not).
It's also worth pointing out, that in addition to this kind of 'folding' of different-but-look-the-same graphemes, in this sort of use case you ABSOLUTELY need to do byte normalization as per UAX#15 http://unicode.org/reports/tr15/ . Probably NFKC for this sort of use case.