News for hackers. Stuff that matters.
Re:Why to normalize Unicode strings by kevin_thibedeau on Wednesday March 13, 2019 @02:22PM (#19381763)
Normalization covers more than just combining characters. CJK full width digits and punctuation can be problematic when you want the canonical forms for pattern matching:

  s = '1 2'

  print(unicodedata.normalize('NFD', s))
  print(unicodedata.normalize('NFC', s))
  print(unicodedata.normalize('NFKC', s))

  1 2
  1 2
  1 2
Re:Why to normalize Unicode strings by lelf on Wednesday March 13, 2019 @02:27PM (#19381829)
It's called collation

  > $*COLLATION.set(:!secondary:!tertiary:!quaternary)
  collation-level => 1, Country => International, Language => None, primary => 1, secondary => 0, tertiary => 0, quaternary => 0

  > '12' coll '12'
  Same