News for hackers. Stuff that matters.
Re:UTF-8000: Unlimited UTF-8 by saghm on Sunday September 20, 2026 @11:17AM (#49776706)
Unless I'm misremembering, even UTF-16 is variable. You need to bump up to UTF-32 to get fixed-width.
Re:UTF-8000: Unlimited UTF-8 by cyphar on Sunday September 20, 2026 @11:33AM (#49776877)
Even better, it's arguably both -- surrogate characters are valid codepoint values so technically UTF-16 is fixed-width but programs need to have special handling for surrogate pairs meaning it is practically variable-width.

Truly the worst of all worlds.

Re:UTF-8000: Unlimited UTF-8 by mafuy on Sunday September 20, 2026 @11:55AM (#49777080)
Correct me if I'm wrong, but I think all kinds of UTF, including 16 and 32, support arbitrary length for a single effective character. This would be because you can stack modifications as long as you like.
Re:UTF-8000: Unlimited UTF-8 by ElectricalUnion on Sunday September 20, 2026 @01:17PM (#49777799)
What you meant by "single effective character" is grapheme clusters. This whole discussion is about variable sized code points.
Re:UTF-8000: Unlimited UTF-8 by Dylan16807 on Sunday September 20, 2026 @01:44PM (#49778082)
The first comment was kind of iffy when it was also talking about buffers and characters, and focusing on code points is mostly a bad focus. It's worth bringing up so nobody thinks fixed width at a single layer is particularly useful, because other layers will still be variable.
Re:UTF-8000: Unlimited UTF-8 by flohofwoe on Monday September 21, 2026 @05:38AM (#49785091)
Yes, UTF-16 is the worst of all alternatives and should be abolished rather sooner than later.

UTF-32 is fixed-width for UNICODE code points, but a single visual character (e.g. a "grapheme cluster") can be built from multiple code points. This is separate from the encoding algorithm though, grapheme clusters are mostly a problem for the high level code working with already decoded text data (text rendering, comparison, sorting etc...).

Re:UTF-8000: Unlimited UTF-8 by nickyvdicarlo on Tuesday September 22, 2026 @02:07PM (#49805627)
I was working on optimizing a UUID scanner recently, and was baffled when I discovered that Java represents all char primitives in UTF-16, regardless of the original encoding.

So even given an ascii String, if you want to do something like

char c = input.charAt(index)

the JVM is going to jump to that index, check whether the character at that index isLatin(), then cast that single byte into to a 2 byte char... every single time.

in the naive solution, omething like 15% of the CPU cycles were spent checking isLatin() over and over again