How UTF-8 Works

Re:How UTF-8 Works by stkdump on Tuesday February 08, 2022 @12:34PM (#30261495)

The current scheme is extensible to 7x6=42 bits (which will probably never be needed). The advantage of the current scheme is that when you read the first byte you know how long the code point is in memory and you have less branching dependencies, i.e. better performance.

EDIT: another huge advantage is that lexicographical comparison/sorting is trivial (usually the ascii version of the code can be reused without modification).

[ Parent | on HN ]

Re:How UTF-8 Works by coldpie on Tuesday February 08, 2022 @01:54PM (#30262748)

> The current scheme is extensible to 7x6=42 bits (which will probably never be needed).

I have printed this out and inserted it into my safe deposit box, so my children's children's children can take it out and have a laugh.

[ Parent | on HN ]

Re:How UTF-8 Works by stkdump on Wednesday February 09, 2022 @01:34PM (#30276417)

Unicode 13 uses 143859 code points. 21 bits can encode more than 10x of that. And there are not that many languages left to add. We are already debating elvish. I am not sure what would warrant extending the coding scheme if we are already on the level of fictional languages today. Also note that many things like flags or emoji skin colors, etc. are now done via grapheme clusters, which conserves code points.

[ Parent | on HN ]

Re:How UTF-8 Works by coldpie on Friday February 11, 2022 @09:46AM (#30302045)

I have stapled your reply to the original printout for extra hilarity! :)

[ Parent | on HN ]

Re:How UTF-8 Works by account42 on Wednesday February 09, 2022 @07:12AM (#30271743)

You can actually extend it indefinitely, 42 bits is just the farthest you can go by trividally extending the bit patterns with the existing logic.

[ Parent | on HN ]

Re:How UTF-8 Works by lkuty on Tuesday February 08, 2022 @12:55PM (#30261840)

like "A Branchless UTF-8 Decoder" at https://nullprogram.com/blog/2017/10/06/

[ Parent | on HN ]