How ASCII lost and unicode won

Re:How ASCII lost and unicode won by drdaeman on Tuesday July 09, 2013 @08:41AM (#6012332)

If we ignore this artificial restriction, we can use arbitrary lengths, turning UTF-8 into varint encoding.

I.e.:

      Oct.1    Oct.2    Oct.3
    11111111 110DDDDD DDDDDDDD (7 octets more)
    \_________/
      10 bits

   ("D"s represent data bits)

This is what'd be necessary to use UTF-8 if we'd need to go beyond the current limits of 0x010FFFF upper bound.

[ Parent | on HN ]

Re:How ASCII lost and unicode won by cygx on Tuesday July 09, 2013 @08:53AM (#6012383)

In your scheme, we'll end up with multiple leading bytes, though.

Also, how do you plan to handle 9 leading bits instead of the 10 from your example? That would make the second byte start with 10, the marker for continuation bytes.

[ Parent | on HN ]

Re:How ASCII lost and unicode won by drdaeman on Tuesday July 09, 2013 @10:09AM (#6012709)

Ouch, you're right. Refreshed my memory of UTF-8 encoding and now it seems it's designed so it can't be made to express arbitrary values. Not sure why I imagined the opposite. My mistake.

[ Parent | on HN ]

Re:How ASCII lost and unicode won by adsr on Tuesday July 09, 2013 @09:35AM (#6012548)

The current limit (0x10FFFF) is the artificial restriction on UTF-8 to match UTF-16, not the upper bound. UTF-8 supports 2^31 code points with it's current encoding. So nothing needs to change until Unicode consists of 2.1 billion code points or so.

In my mind that will only happen as a result of extreme carelessness.

[ Parent | on HN ]