Re:Brian Kernighan adds Unicode support to Awk
by tialaramex on Saturday August 20, 2022 @03:09PM
(#32534424)
Right, somebody else might have actual metrics but I'd have guessed actual regular expression patterns are split something like:
90% Only care about ASCII, thus individual bytes in UTF-8, and so UTF-32 just wastes memory
1% Care about individual code points, but spread over multiple bytes (e.g. the double dagger ‡), UTF-32 is perfect
9% Care about multiple code points (to form e.g. a Flag, or é written in combining form, or two women kissing) and so UTF-32 doesn't really help again
Re:Brian Kernighan adds Unicode support to Awk
by bmn__ on Sunday August 21, 2022 @05:26AM
(#32538590)
Applications and text-processing libraries are free use an internal text encoding or data structure, if you will, which does not suffer from the drawbacks. Conversions to encodings suitable for data exchange (e.g. UTF-8) are performed at the I/O boundaries.
Re:Brian Kernighan adds Unicode support to Awk
by tialaramex on Sunday August 21, 2022 @04:11PM
(#32543935)
Sure, and if stuff lives inside your "application and text-processing library" for long enough and performs enough of the thusly optimised actions without leaving, you might even amortize the cost of the work you did at the edges.
But probably not.
Re:Brian Kernighan adds Unicode support to Awk
by layer8 on Saturday August 20, 2022 @06:03PM
(#32535554)
Another factor is that nowadays machine code execution is much faster than memory accesses, so the trade-off of requiring more program logic to process a more compact format makes a lot of sense.
I think these days people don't bother with UTF-32 too much because it's not even like you have a clean "one 32bit int, one character" relation anyway since some characters can be built from multiple codepoints. Since generally most code manipulating character strings are interested in characters and not codepoints, UTF-32 is effectively a variable-length encoding too...