Brian Kernighan adds Unicode support to Awk

Re:Brian Kernighan adds Unicode support to Awk by tialaramex on Saturday August 20, 2022 @03:09PM (#32534424)

Right, somebody else might have actual metrics but I'd have guessed actual regular expression patterns are split something like:

90% Only care about ASCII, thus individual bytes in UTF-8, and so UTF-32 just wastes memory

1% Care about individual code points, but spread over multiple bytes (e.g. the double dagger ‡), UTF-32 is perfect

9% Care about multiple code points (to form e.g. a Flag, or é written in combining form, or two women kissing) and so UTF-32 doesn't really help again

[ Parent | on HN ]

Re:Brian Kernighan adds Unicode support to Awk by bmn__ on Sunday August 21, 2022 @05:26AM (#32538590)

Applications and text-processing libraries are free use an internal text encoding or data structure, if you will, which does not suffer from the drawbacks. Conversions to encodings suitable for data exchange (e.g. UTF-8) are performed at the I/O boundaries.

[ Parent | on HN ]

Re:Brian Kernighan adds Unicode support to Awk by tialaramex on Sunday August 21, 2022 @04:11PM (#32543935)

Sure, and if stuff lives inside your "application and text-processing library" for long enough and performs enough of the thusly optimised actions without leaving, you might even amortize the cost of the work you did at the edges.

But probably not.

[ Parent | on HN ]