Re:Why to normalize Unicode strings
by happytoexplain on Wednesday March 13, 2019 @02:03PM
(#19381552)
Swift is another major language that has correctly solved this problem in this way - i.e. not representing/operating on strings as though they were naive arrays of bytes or code points - but rather as though they were arrays of characters, which Unicode thoroughly and intuitively defines in the same way that humans think about characters.
Swift is headed in the right direction.
But I think Perl6 is the only language that can do this magic:
> 'Déjà vu' ~~ /:ignorecase:ignoremark deja \s vu/
「Déjà vu」> characters, which Unicode thoroughly and intuitively defines in the same way that humans think about characters
"Character" is a somewhat vague term, and Unicode prefers to use more specific terms like "code unit", "code point", "abstract character", etc.
In this case I think you may be referring to grapheme clusters, which come closer to how "humans think about characters" than Unicode abstract characters, which are building blocks of the technical encoding standard but in some cases don't really match a human concept of a graphical element of a writing system.
See also “Characters” and Grapheme Clusters in section 2.11 of https://www.unicode.org/versions/Unicode12.0.0/ch02.pdf, for example.
Re:Why to normalize Unicode strings
by happytoexplain on Wednesday March 13, 2019 @03:32PM
(#19382694)
Oops - you're right. I'm using the term "character" for both the intuitive and documented definition, but the documented term I'm referring to is actually grapheme cluster.
Re:Why to normalize Unicode strings
by kevin_thibedeau on Wednesday March 13, 2019 @02:22PM
(#19381763)
Normalization covers more than just combining characters. CJK full width digits and punctuation can be problematic when you want the canonical forms for pattern matching:
s = '1 2'
print(unicodedata.normalize('NFD', s))
print(unicodedata.normalize('NFC', s))
print(unicodedata.normalize('NFKC', s))
1 2
1 2
1 2It's called collation
> $*COLLATION.set(:!secondary:!tertiary:!quaternary)
collation-level => 1, Country => International, Language => None, primary => 1, secondary => 0, tertiary => 0, quaternary => 0
> '12' coll '12'
Same
PS: WTF? HN strips emojis :/ (and does it incorrectly when they are emoji sequences).