Re:The curious case of the disappearing Polish S (2015)
by dhosek on Friday May 21, 2021 @06:06PM
(#27240568)
It's surprising how much software (mostly on Windows) doesn't properly handle Unicode in 2021. With something working in Unicode, it's not that big of a deal to both handle letters like Ł or ż and also to run normalization on text strings so that you can (if desired) treat Łódź and Lodz as identical (e.g., for text searching).
[ Parent
| on HN ]
Re:The curious case of the disappearing Polish S (2015)
by bobbylarrybobby on Friday May 21, 2021 @06:36PM
(#27240829)
Just a note that normalization is not the same as diacritic insensitivity. Normalization is the process by which strings that are semantically equivalent (by some standard), yet have different underlying byte sequences, are transformed to have the same underlying byte sequence. For instance, replacing “e, combining acute accent” with “e with acute accent”.
Re:The curious case of the disappearing Polish S (2015)
by dhosek on Friday May 21, 2021 @10:16PM
(#27242664)
Yes, I should clarify that the process is to normalize to letter+combining accent and then strip out the accents, e.g., in Java:
Normalizer.normalize(text, Normalizer.Form.NFD).replaceAll("\\p{M}", "");
(I hope I got this right—I adapted it from some code which actually strips out some other bits of text to turn, e.g., F. Janaček into FJanacek.)