News for hackers. Stuff that matters.

A comment on The future of 32-bit support in the kernel. Parent comment

Re:The future of 32-bit support in the kernel by kbolino on Tuesday September 02, 2025 @12:33PM (#45105392)
In order to implement grapheme cluster segmentation, you have to start with a sequence of Unicode scalars. In practice, that means a sequence of 32-bit integers, which is UTF-32 in all but name. It's not a good interchange format, but it is a necessary intermediate/internal format.

There's also the problem that grapheme cluster boundaries change over time. Unicode has become a true mess.

[ Parent | on HN ]
Re:The future of 32-bit support in the kernel by mort96 on Tuesday September 02, 2025 @12:43PM (#45105564)
Yeah, you need some kind of sequence of Unicode scalars. But there's no reason for that sequence to be "a contiguous chunk of memory filled with 32-bit ints" (aka a UTF-32 string); it can just as well be an iterator which operates on an in-memory UTF-8 string and produces code points.
[ Parent | on HN ]