Re:The future of 32-bit support in the kernel
by adgjlsfhk1 on Monday September 01, 2025 @08:51PM
(#45098003)
UTF-16 is the worst of all worlds. Either use UTF32 where code-points are fixed, or if you care about space efficiency use UTF8
[ Parent
| on HN ]
Re:The future of 32-bit support in the kernel
by mort96 on Tuesday September 02, 2025 @03:22AM
(#45100058)
UTF-32 is arguably even more worst of all worlds. You don't get fixed-size units in any meaningful way. Yes you have fixed sized code points, but those aren't the "units" you care about; you still have variable size grapheme clusters, so you still can't do things like reversing a string or splitting a string at an arbitrary index or anything else like that. Yet it consumes twice the space of UTF-16 for almost everything, and four times the space of UTF-8 for many things.
UTF-32 is the worst of all worlds. UTF-16 has the teeny tiny advantage that pure Chinese text takes a bit less space in UTF-16 than UTF-8 (typically irrelevant because that advantage is outweighed by the fact that the markup surrounding the text takes more space). UTF-8 is the best option for pretty much everything.
As a consequence, never use UTF-32, only use UTF-16 where necessary due to backwards compatibility, always use UTF-8 where possible.
Re:The future of 32-bit support in the kernel
by kbolino on Tuesday September 02, 2025 @12:33PM
(#45105392)
In order to implement grapheme cluster segmentation, you have to start with a sequence of Unicode scalars. In practice, that means a sequence of 32-bit integers, which is UTF-32 in all but name. It's not a good interchange format, but it is a necessary intermediate/internal format.
There's also the problem that grapheme cluster boundaries change over time. Unicode has become a true mess.
Re:The future of 32-bit support in the kernel
by mort96 on Tuesday September 02, 2025 @12:43PM
(#45105564)
Yeah, you need some kind of sequence of Unicode scalars. But there's no reason for that sequence to be "a contiguous chunk of memory filled with 32-bit ints" (aka a UTF-32 string); it can just as well be an iterator which operates on an in-memory UTF-8 string and produces code points.