Re:Ruby 1.9 made me remember how I hate the concept of encod...
by jerf on Thursday May 13, 2010 @02:12PM
(#1345085)
"This, to me, is the fundamental flaw. By now we should be able to have a single encoding that takes up as little space as possible while supporting every known character, and leaving room for more."
That's pretty much UTF-8. If you're going to stuff everything into one encoding, you are going to have to make tradeoffs.
See also UCS-4, which is simply "throw 32 bits at every character". Nobody uses it because it makes everything pretty big. (At least a 3-byte CJK character tends to mean more on average than a single English character.)
If you haven't already, take some time to read over the Unicode standards. It is very enlightening. This is especially true if you want to make the "one true encoding"; gotta know what the bar is to beat, right? There's way more to Unicode than just "Here's a catalog of all possible characters and here's numbers for them all", and there are reasons why there's way more than that.
This, to me, is the fundamental flaw. By now we should be able to have a single encoding that takes up as little space as possible while supporting every known character, and leaving room for more. Most machines now are at least 32-bit... that's almost 4.3 billion characters. Surely there are less than that in the world?
One of my "Grand Lifetime Projects" is to build a new programming language and a new OS built with it. Part of that will certainly be handling strings in an efficient way, both in terms of computer and programmer time. I have some ideas swirling around for creating One True Encoding that allows for extensibility.