Re:Ruby 1.9 made me remember how I hate the concept of encod...
by jerf on Thursday May 13, 2010 @02:12PM
(#1345085)
"This, to me, is the fundamental flaw. By now we should be able to have a single encoding that takes up as little space as possible while supporting every known character, and leaving room for more."
That's pretty much UTF-8. If you're going to stuff everything into one encoding, you are going to have to make tradeoffs.
See also UCS-4, which is simply "throw 32 bits at every character". Nobody uses it because it makes everything pretty big. (At least a 3-byte CJK character tends to mean more on average than a single English character.)
If you haven't already, take some time to read over the Unicode standards. It is very enlightening. This is especially true if you want to make the "one true encoding"; gotta know what the bar is to beat, right? There's way more to Unicode than just "Here's a catalog of all possible characters and here's numbers for them all", and there are reasons why there's way more than that.
That's pretty much UTF-8. If you're going to stuff everything into one encoding, you are going to have to make tradeoffs.
See also UCS-4, which is simply "throw 32 bits at every character". Nobody uses it because it makes everything pretty big. (At least a 3-byte CJK character tends to mean more on average than a single English character.)
If you haven't already, take some time to read over the Unicode standards. It is very enlightening. This is especially true if you want to make the "one true encoding"; gotta know what the bar is to beat, right? There's way more to Unicode than just "Here's a catalog of all possible characters and here's numbers for them all", and there are reasons why there's way more than that.