Validating UTF-8 strings using as little as 0.7 cycles per byte

Re:Validating UTF-8 strings using as little as 0.7 cycles pe... by kizer on Wednesday May 16, 2018 @07:09PM (#17087191)

I did not know that Unicode itself was variable length. That alone nullifies motivation to use codepoint arrays. Why would they do that??

[ Parent | on HN ]

Re:Validating UTF-8 strings using as little as 0.7 cycles pe... by masklinn on Thursday May 17, 2018 @03:57AM (#17089594)

To limit combinatoric explosion. Without combining codepoints, you need a version of each base symbol for each combination of modifier. And while latin scripts usually limit themselves to one diacritic per base[0], other scripts (hebrew or arabic come to mind) can pile up multiple diacritics.

[0] not that that's always the case, the navajo alphabet can have both ogonek and acute on the same latin base character

[ Parent | on HN ]

Re:Validating UTF-8 strings using as little as 0.7 cycles pe... by kizer on Thursday May 17, 2018 @10:20AM (#17091422)

I see... speaks to my ignorance on language.

[ Parent | on HN ]