Re:UTF-8000: Unlimited UTF-8
by saghm on Sunday September 20, 2026 @11:17AM
(#49776706)
Unless I'm misremembering, even UTF-16 is variable. You need to bump up to UTF-32 to get fixed-width.
[ Parent
| on HN ]
Even better, it's arguably both -- surrogate characters are valid codepoint values so technically UTF-16 is fixed-width but programs need to have special handling for surrogate pairs meaning it is practically variable-width.
Truly the worst of all worlds.
Correct me if I'm wrong, but I think all kinds of UTF, including 16 and 32, support arbitrary length for a single effective character.
This would be because you can stack modifications as long as you like.
Re:UTF-8000: Unlimited UTF-8
by ElectricalUnion on Sunday September 20, 2026 @01:17PM
(#49777799)
What you meant by "single effective character" is grapheme clusters. This whole discussion is about variable sized code points.
Re:UTF-8000: Unlimited UTF-8
by Dylan16807 on Sunday September 20, 2026 @01:44PM
(#49778082)
The first comment was kind of iffy when it was also talking about buffers and characters, and focusing on code points is mostly a bad focus. It's worth bringing up so nobody thinks fixed width at a single layer is particularly useful, because other layers will still be variable.
Yes, UTF-16 is the worst of all alternatives and should be abolished rather sooner than later.
UTF-32 is fixed-width for UNICODE code points, but a single visual character (e.g. a "grapheme cluster") can be built from multiple code points. This is separate from the encoding algorithm though, grapheme clusters are mostly a problem for the high level code working with already decoded text data (text rendering, comparison, sorting etc...).
Re:UTF-8000: Unlimited UTF-8
by nickyvdicarlo on Tuesday September 22, 2026 @02:07PM
(#49805627)
I was working on optimizing a UUID scanner recently, and was baffled when I discovered that Java represents all char primitives in UTF-16, regardless of the original encoding.
So even given an ascii String, if you want to do something like
char c = input.charAt(index)
the JVM is going to jump to that index, check whether the character at that index isLatin(), then cast that single byte into to a 2 byte char... every single time.
in the naive solution, omething like 15% of the CPU cycles were spent checking isLatin() over and over again