News for hackers. Stuff that matters.
Re:The history of UTF-8 as told by Rob Pike (2003) by morelisp on Thursday April 08, 2021 @07:17AM (#26737398)
The 5-6 byte variants (and also 4 at the time) exist because of the need to round-trip UCS surrogate pairs through UTF-8, no? That's what I assume the "political reasons" are...
Re:The history of UTF-8 as told by Rob Pike (2003) by Sniffnoy on Thursday April 08, 2021 @11:02AM (#26739651)
Others have already answered why surrogate pairs are irrelevant (and not UCS), but I think it's worth saying what the probable actual reason for 5-6 byte variants was. Remember that UCS and Unicode were at this point still two separate things; Unicode was supposed to be 16-bit (and later it got expanded, causing the whole surrogates mess), while UCS was supposed to be 31-bit. I assume the 5-6 byte variants were for UCS (back before it got merged with Unicode).
Re:The history of UTF-8 as told by Rob Pike (2003) by ChrisSD on Thursday April 08, 2021 @07:48AM (#26737603)
Surrogate pairs are only in UTF-16 so as to encode code points that require more than 16 bits. UTF-8 has no need of them because it's already a variable width encoding.

If there were no code points larger than 16 bits then UTF-8 would only need a maximum of 3 bytes per code point and UTF-16 wouldn't need surrogate pairs. Well actually UTF-16 probably wouldn't exist at all because UCS-2 would have been enough for everybody.

Re:The history of UTF-8 as told by Rob Pike (2003) by cryptonector on Thursday April 08, 2021 @04:50PM (#26743592)
No. They exist to encode 31 bits of codepoint space, but later the UC decided to limit the codepoint space to only 21 bits because that is what UTF-16 is limited to, and then UTF-8 no longer needed to support sequences of 5 and 6 bytes.
Re:The history of UTF-8 as told by Rob Pike (2003) by 0x0 on Thursday April 08, 2021 @07:29AM (#26737492)
I don't think so? Aren't UCS surrogate pairs at most 16bit each by their very purpose? Also, >16bit unicode code points came much later, I believe, in Unicode 2.0 in 1996 according to Wikipedia (vs UTF-8 which is from around 1992)