Re:Validating UTF-8 strings using as little as 0.7 cycles pe...
by kizer on Wednesday May 16, 2018 @07:09PM
(#17087191)
I did not know that Unicode itself was variable length. That alone nullifies motivation to use codepoint arrays. Why would they do that??
Re:Validating UTF-8 strings using as little as 0.7 cycles pe...
by masklinn on Thursday May 17, 2018 @03:57AM
(#17089594)
To limit combinatoric explosion. Without combining codepoints, you need a version of each base symbol for each combination of modifier. And while latin scripts usually limit themselves to one diacritic per base[0], other scripts (hebrew or arabic come to mind) can pile up multiple diacritics.
[0] not that that's always the case, the navajo alphabet can have both ogonek and acute on the same latin base character
Re:Validating UTF-8 strings using as little as 0.7 cycles pe...
by kizer on Thursday May 17, 2018 @10:20AM
(#17091422)
I see... speaks to my ignorance on language.
Re:Validating UTF-8 strings using as little as 0.7 cycles pe...
by bumblebritches5 on Wednesday May 16, 2018 @02:48PM
(#17085273)
Have you ever written your own unicode library, or are you just parroting what you've heard?
I have written my own unicode library, normalizing strings as UTF-32 is MUCH MUCH MUCH easier than trying to do the same in UTF-8 or UTF-16.
not to mention casefolding, and it also allows you to design better interfaces, so that the actual algorithm only needs to be implemented once.
Re:Validating UTF-8 strings using as little as 0.7 cycles pe...
by greglindahl on Wednesday May 16, 2018 @06:41PM
(#17087045)
Sounds like your library is the poster-child for:
> it blows up memory usage for ASCII and BMP (4 bytes per codepoint versus 1~3)
No. I think the implementor of Factor's unicode support originally did that but it turns out to not be useful:
* it blows up memory usage for ASCII and BMP (4 bytes per codepoint versus 1~3)
* this also has impact on CPU caches, lowering the average amount of data you can fit in your cache and work on
* it requires a complete conversion of incoming ascii and utf8 data (which only get more and more common as time goes on) rather than just a validation
* and because Unicode itself is variable-length (combining codepoints) it's not actually helpful when you're trying to properly manipulate unicode data
The only "advantage" of representing strings as codepoint arrays is that you get O(1) access to codepoints which is a terrible idea you should not encourage.
UTF-32 internal encoding makes some manipulations very slightly easier, but not enough to matter in the long run, and it encourages bad habits. If you don't need the O(1) access thing for backwards compatibility reasons, don't bother.