Re:Brian Kernighan adds Unicode support to Awk
by tialaramex on Saturday August 20, 2022 @05:48PM
(#32535431)
Unicode isn't a storage encoding and so yeah, Notepad shouldn't do that. However Unicode does encode essentially all extant human writing systems into integers called "code points" between zero and 0x10FFFF. The Latin "capital A" is 65 for example.
However you'd probably like to store something more compact than, say, JSON arrays of integers. So there are also a bunch of encodings which turn the integers into bytes. These encodings would work for any integers, but they make most sense to encode Unicode's code points. UTF-8 turns each code point into 1-4 bytes, a pair of UTF-16 encodings turns them into one or two "code units" each of 2 bytes either little or big endian. And UTF-32 just encodes them as native 32-bit integers but again either little or big endian.
However you'd probably like to store something more compact than, say, JSON arrays of integers. So there are also a bunch of encodings which turn the integers into bytes. These encodings would work for any integers, but they make most sense to encode Unicode's code points. UTF-8 turns each code point into 1-4 bytes, a pair of UTF-16 encodings turns them into one or two "code units" each of 2 bytes either little or big endian. And UTF-32 just encodes them as native 32-bit integers but again either little or big endian.