News for hackers. Stuff that matters.
Re:UTF-8 – "The most elegant hack" by ygra on Sunday September 29, 2013 @04:27AM (#6464539)
There are multiple factors at play here and it ultimately comes down to what you need:

If you to access code points, then UTF-32 gives you a nice fixed-width (albeit quite large) encoding to work with. Some applications or libraries may use UTF-32 internally to speed things up.

If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points: e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾ (this probably doesn't even render properly).

Sadly, most APIs use either UTF-8 or UTF-16 and consider code units the most important entity, not code points. And that's inexcusable and frankly a mess, because I can count the number of times I needed to access a code unit instead of a code point on zero hands.

Re:UTF-8 – "The most elegant hack" by tripzilch on Sunday September 29, 2013 @06:42AM (#6464761)
> If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points:

> e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾

As how many characters am I supposed to perceive this, anyway? Some of those tiny squiggles are in fact letters (I mean they look like letters, I see an e,m,u,o,i and r in there). What if you'd make a crossword this way?

Re:UTF-8 – "The most elegant hack" by dbaupp on Sunday September 29, 2013 @07:34AM (#6464861)
It says "example" with a pile of combining characters (77 to be precise): http://i.imgur.com/acyHBkR.png
Re:UTF-8 – "The most elegant hack" by thaumasiotes on Sunday September 29, 2013 @09:16AM (#6465089)
I've experienced crosswords where e.g. the word going across uses a diacritic on a particular letter ("élan") and the word going down doesn't. It drives me crazy.
Re:UTF-8 – "The most elegant hack" by riffraff on Sunday September 29, 2013 @01:55PM (#6465932)
FYI, that is the way crosswords work in italian, accents are considered irrelevant.
Re:UTF-8 – "The most elegant hack" by iv_08 on Sunday September 29, 2013 @06:04AM (#6464705)
For index based operations code units are more useful than code points.

a = s.indexOf("e̬̱ͤ̈́̿̂̅́͟")

b = s.indexOf("m̛̮̩͙ͦ͗͆̄̋́̄")

s[a:b]

This operation is fast with code unit indices and slow with code point indices.

Re:UTF-8 – "The most elegant hack" by Dylan16807 on Sunday September 29, 2013 @01:39PM (#6465865)
If you use stream-safe normalization rules, then you have a worst-case limit of 30 combining code points in a row. You could even make an array of 128 byte single-character-strings if you wanted.
Re:UTF-8 – "The most elegant hack" by Flimm on Sunday September 29, 2013 @05:10AM (#6464610)
Please define what you mean by code point and code unit.
Re:UTF-8 – "The most elegant hack" by ygra on Sunday September 29, 2013 @07:47AM (#6464890)
I mean the same thing that Unicode uses these terms for.

Code Unit. The minimal bit combination that can represent a unit of encoded text for processing or interchange. The Unicode Standard uses 8-bit code units in the UTF-8 encoding form, 16-bit code units in the UTF-16 encoding form, and 32-bit code units in the UTF-32 encoding form.

Code Point. (1) Any value in the Unicode codespace; that is, the range of integers from 0 to 10FFFF16. (See definition D10 in Section 3.4, Characters and Encoding.) Not all code points are assigned to encoded characters. See code point type. (2) A value, or position, for a character, in any coded character set.

From http://www.unicode.org/glossary/