News for hackers. Stuff that matters.
Re:UTF-8 – "The most elegant hack" by runn1ng on Saturday September 28, 2013 @07:50PM (#6463525)
UTF8 != unicode.

Unicode is a standard, that maps the giant number of letters to unique numbers.

UTF8 (and UTF16, UTF32) are various encodings of that. Unicode itself is not an encoding; unicode just says "letter 'č' is 269"; UTF-8 says "We will represent it as two bytes, first is C4, second is 8D".

UTF8 is indeed variable width; UTF16 and UTF32 are, AFAIK, fixed width. Both use unicode standard.

Re:UTF-8 – "The most elegant hack" by Stratoscope on Saturday September 28, 2013 @08:15PM (#6463592)
Even if we're talking about code points and not characters, UTF-16 is variable width.

UTF-8 is variable width using 8, 16, 24, or 32 bits per code point.

UCS-2 is fixed width using 16 bits per code point.

UTF-16 is variable width using 16 or 32 bits per code point. It's an extension of the fixed width UCS-2 format: every valid 16-bit UCS-2 code point is also a valid UTF-16 code point, but UTF-16 also includes 32-bit code points.

UTF-32 is fixed width using 32 bits per code point.

Re:UTF-8 – "The most elegant hack" by alister on Saturday September 28, 2013 @10:28PM (#6463870)
UTF-8 is variable width using 8, 16, 24, or 32 bits per code point.

I wondered why you didn't write that "UTF-8 is variable width using 8, 16, 24, 32, 40, or 48 bits per code point", because that's how Prosser/Thompson/Pike's UTF-8 idea was proposed, as shown in the "most elegant hack" article.

And here's the answer from Wikipedia: In November 2003 UTF-8 was restricted by RFC 3629 to end at U+10FFFF, in order to match the constraints of the UTF-16 character encoding. This removed all 5- and 6-byte sequences, and about half of the 4-byte sequences.

Even though they eliminated the 5th and 6th bytes, UTF-8 can still represent all of Unicode's 1,114,112 code points.

I'm mentioning this because others probably are wondering the same thing.

Re:UTF-8 – "The most elegant hack" by option_greek on Sunday September 29, 2013 @03:44AM (#6464455)
How sad. One defective format limits another great format.
Re:UTF-8 – "The most elegant hack" by ygra on Sunday September 29, 2013 @04:21AM (#6464527)
We don't see the code point space run out anytime soon this time, even with questionable additions like Emoji. It took us 20 years to get to the current state and character encoding rate has slowed down significantly since the early days (mostly because most things are covered already).
Re:UTF-8 – "The most elegant hack" by btilly on Saturday September 28, 2013 @11:14PM (#6463954)
UTF8 is indeed variable width; UTF16 and UTF32 are, AFAIK, fixed width. Both use unicode standard.

Your knowledge is wrong.

As of 2012, there are 110,182 assigned Unicode characters, and 137,468 more which have been reserved for private use.

With 16 bits there are only 65,536 possible values, and so UTF-16 has to be variable width. (Though it was originally intended to be fixed width. A misconception unfortunately embedded in the design of languages from the time such as Java.)

Re:UTF-8 – "The most elegant hack" by pjscott on Saturday September 28, 2013 @07:57PM (#6463538)
What the comment you're replying to is saying that glyphs may be represented by more than one code point. So sure, UTF32 has a fixed width for each code point, but it's not quite so clear how useful that is.
Re:UTF-8 – "The most elegant hack" by runn1ng on Saturday September 28, 2013 @08:01PM (#6463552)
Oh, in that case I misunderstood parent. Sorry about that.
Re:UTF-8 – "The most elegant hack" by gsnedders on Saturday September 28, 2013 @09:44PM (#6463771)
The Unicode standard nowadays defines UTF-8, UTF-16, and UTF-32. It speaks of "characters" which are assigned (i.e., mapped) to "code points" (i.e., U+0000 to U+10FFFF) and character encodings which encode scalar values (all code points except the surrogate ranges, therefore including all characters).