News for hackers. Stuff that matters.
Re:UTF-8 is a brilliant design by spankalee on Friday September 12, 2025 @05:23PM (#45226925)
Wouldn't you only need to read backwards at most 3 bytes to see if you were currently at a continuation byte? With a max multi-byte size of 4 bytes, if you don't see a multi-byte start character by then you would know it's a single-byte char.

I wonder if a reason is similar though: error recovery when working with libraries that aren't UTF-8 aware. If you slice naively slice an array of UTF-8 bytes, a UTf-8 aware library can ignore malformed leading and trailing bytes and get some reasonable string out of it.

Re:UTF-8 is a brilliant design by Sharlin on Friday September 12, 2025 @09:33PM (#45228565)
It’s not always possible to read backwards.
Re:UTF-8 is a brilliant design by Dylan16807 on Saturday September 13, 2025 @01:02AM (#45229448)
Okay so you seek by 3 less bytes.

Or you accept that if you're randomly losing chunks, you might lose an extra 3 bytes.

The real problem is that seeking a few bytes won't work with EMBL. If continuation bytes store 8 payload bits, you can get into a situation where every single byte could be interpreted as a multi-byte start character and there are 2 or 3 possible messages that never converge.

Re:UTF-8 is a brilliant design by Sharlin on Saturday September 13, 2025 @09:21AM (#45231899)
The point is that you don’t have a "seek" operation available. You are given a bytestream and aren’t told if you’re at the start, in a valid position between code points, or in the middle of a code point. UTF-8’s self-synchronizing property means that by reading a single byte you immediately know if you’re in the middle of a code point, and that by reading and discarding at most two additional bytes you’re synchronized and can start/return decoding. That wouldn’t be possible if continuation bytes used all the bits for payload.
Re:UTF-8 is a brilliant design by Dylan16807 on Saturday September 13, 2025 @02:08PM (#45234083)
Yes, the point is being able to synchronize.

But it doesn't matter if it takes 1 byte or 3 bytes to synchronize. And being unable to read backwards is not a problem.

(EMBL doesn't synchronize in three bytes but other encodings do.)