Re:The curious case of the disappearing Polish S (2015)
by bobbylarrybobby on Friday May 21, 2021 @06:36PM
(#27240829)
Just a note that normalization is not the same as diacritic insensitivity. Normalization is the process by which strings that are semantically equivalent (by some standard), yet have different underlying byte sequences, are transformed to have the same underlying byte sequence. For instance, replacing “e, combining acute accent” with “e with acute accent”.
[ Parent
| on HN ]
Re:The curious case of the disappearing Polish S (2015)
by dhosek on Friday May 21, 2021 @10:16PM
(#27242664)
Yes, I should clarify that the process is to normalize to letter+combining accent and then strip out the accents, e.g., in Java:
Normalizer.normalize(text, Normalizer.Form.NFD).replaceAll("\\p{M}", "");
(I hope I got this right—I adapted it from some code which actually strips out some other bits of text to turn, e.g., F. Janaček into FJanacek.)