News for hackers. Stuff that matters.
Re:UTF-8 – "The most elegant hack" by sopooneo on Saturday September 28, 2013 @07:12PM (#6463450)
Please correct me if I'm wrong, but you don't really get fixed width in anyway do you? Because unicode itself is not fixed width. There are situations where one character on the screen is represented by two consecutive unicode code points.
Re:UTF-8 – "The most elegant hack" by plorkyeran on Saturday September 28, 2013 @07:46PM (#6463503)
Or a lot more than two, or even multiple characters for a single code point.

UTF-32 appears to simplify things a lot, but in practice O(1) indexing of code points and simple iteration over code points is not actually all that useful, because code points are not characters.

Re:UTF-8 – "The most elegant hack" by ygra on Sunday September 29, 2013 @04:27AM (#6464539)
There are multiple factors at play here and it ultimately comes down to what you need:

If you to access code points, then UTF-32 gives you a nice fixed-width (albeit quite large) encoding to work with. Some applications or libraries may use UTF-32 internally to speed things up.

If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points: e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾ (this probably doesn't even render properly).

Sadly, most APIs use either UTF-8 or UTF-16 and consider code units the most important entity, not code points. And that's inexcusable and frankly a mess, because I can count the number of times I needed to access a code unit instead of a code point on zero hands.

Re:UTF-8 – "The most elegant hack" by tripzilch on Sunday September 29, 2013 @06:42AM (#6464761)
> If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points:

> e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾

As how many characters am I supposed to perceive this, anyway? Some of those tiny squiggles are in fact letters (I mean they look like letters, I see an e,m,u,o,i and r in there). What if you'd make a crossword this way?

Re:UTF-8 – "The most elegant hack" by dbaupp on Sunday September 29, 2013 @07:34AM (#6464861)
It says "example" with a pile of combining characters (77 to be precise): http://i.imgur.com/acyHBkR.png
Re:UTF-8 – "The most elegant hack" by thaumasiotes on Sunday September 29, 2013 @09:16AM (#6465089)
I've experienced crosswords where e.g. the word going across uses a diacritic on a particular letter ("élan") and the word going down doesn't. It drives me crazy.
Re:UTF-8 – "The most elegant hack" by riffraff on Sunday September 29, 2013 @01:55PM (#6465932)
FYI, that is the way crosswords work in italian, accents are considered irrelevant.
Re:UTF-8 – "The most elegant hack" by iv_08 on Sunday September 29, 2013 @06:04AM (#6464705)
For index based operations code units are more useful than code points.

a = s.indexOf("e̬̱ͤ̈́̿̂̅́͟")

b = s.indexOf("m̛̮̩͙ͦ͗͆̄̋́̄")

s[a:b]

This operation is fast with code unit indices and slow with code point indices.

Re:UTF-8 – "The most elegant hack" by Dylan16807 on Sunday September 29, 2013 @01:39PM (#6465865)
If you use stream-safe normalization rules, then you have a worst-case limit of 30 combining code points in a row. You could even make an array of 128 byte single-character-strings if you wanted.
Re:UTF-8 – "The most elegant hack" by Flimm on Sunday September 29, 2013 @05:10AM (#6464610)
Please define what you mean by code point and code unit.
Re:UTF-8 – "The most elegant hack" by ygra on Sunday September 29, 2013 @07:47AM (#6464890)
I mean the same thing that Unicode uses these terms for.

Code Unit. The minimal bit combination that can represent a unit of encoded text for processing or interchange. The Unicode Standard uses 8-bit code units in the UTF-8 encoding form, 16-bit code units in the UTF-16 encoding form, and 32-bit code units in the UTF-32 encoding form.

Code Point. (1) Any value in the Unicode codespace; that is, the range of integers from 0 to 10FFFF16. (See definition D10 in Section 3.4, Characters and Encoding.) Not all code points are assigned to encoded characters. See code point type. (2) A value, or position, for a character, in any coded character set.

From http://www.unicode.org/glossary/

Re:UTF-8 – "The most elegant hack" by runn1ng on Saturday September 28, 2013 @07:50PM (#6463525)
UTF8 != unicode.

Unicode is a standard, that maps the giant number of letters to unique numbers.

UTF8 (and UTF16, UTF32) are various encodings of that. Unicode itself is not an encoding; unicode just says "letter 'č' is 269"; UTF-8 says "We will represent it as two bytes, first is C4, second is 8D".

UTF8 is indeed variable width; UTF16 and UTF32 are, AFAIK, fixed width. Both use unicode standard.

Re:UTF-8 – "The most elegant hack" by Stratoscope on Saturday September 28, 2013 @08:15PM (#6463592)
Even if we're talking about code points and not characters, UTF-16 is variable width.

UTF-8 is variable width using 8, 16, 24, or 32 bits per code point.

UCS-2 is fixed width using 16 bits per code point.

UTF-16 is variable width using 16 or 32 bits per code point. It's an extension of the fixed width UCS-2 format: every valid 16-bit UCS-2 code point is also a valid UTF-16 code point, but UTF-16 also includes 32-bit code points.

UTF-32 is fixed width using 32 bits per code point.

Re:UTF-8 – "The most elegant hack" by alister on Saturday September 28, 2013 @10:28PM (#6463870)
UTF-8 is variable width using 8, 16, 24, or 32 bits per code point.

I wondered why you didn't write that "UTF-8 is variable width using 8, 16, 24, 32, 40, or 48 bits per code point", because that's how Prosser/Thompson/Pike's UTF-8 idea was proposed, as shown in the "most elegant hack" article.

And here's the answer from Wikipedia: In November 2003 UTF-8 was restricted by RFC 3629 to end at U+10FFFF, in order to match the constraints of the UTF-16 character encoding. This removed all 5- and 6-byte sequences, and about half of the 4-byte sequences.

Even though they eliminated the 5th and 6th bytes, UTF-8 can still represent all of Unicode's 1,114,112 code points.

I'm mentioning this because others probably are wondering the same thing.

Re:UTF-8 – "The most elegant hack" by option_greek on Sunday September 29, 2013 @03:44AM (#6464455)
How sad. One defective format limits another great format.
Re:UTF-8 – "The most elegant hack" by ygra on Sunday September 29, 2013 @04:21AM (#6464527)
We don't see the code point space run out anytime soon this time, even with questionable additions like Emoji. It took us 20 years to get to the current state and character encoding rate has slowed down significantly since the early days (mostly because most things are covered already).
Re:UTF-8 – "The most elegant hack" by btilly on Saturday September 28, 2013 @11:14PM (#6463954)
UTF8 is indeed variable width; UTF16 and UTF32 are, AFAIK, fixed width. Both use unicode standard.

Your knowledge is wrong.

As of 2012, there are 110,182 assigned Unicode characters, and 137,468 more which have been reserved for private use.

With 16 bits there are only 65,536 possible values, and so UTF-16 has to be variable width. (Though it was originally intended to be fixed width. A misconception unfortunately embedded in the design of languages from the time such as Java.)

Re:UTF-8 – "The most elegant hack" by pjscott on Saturday September 28, 2013 @07:57PM (#6463538)
What the comment you're replying to is saying that glyphs may be represented by more than one code point. So sure, UTF32 has a fixed width for each code point, but it's not quite so clear how useful that is.
Re:UTF-8 – "The most elegant hack" by runn1ng on Saturday September 28, 2013 @08:01PM (#6463552)
Oh, in that case I misunderstood parent. Sorry about that.
Re:UTF-8 – "The most elegant hack" by gsnedders on Saturday September 28, 2013 @09:44PM (#6463771)
The Unicode standard nowadays defines UTF-8, UTF-16, and UTF-32. It speaks of "characters" which are assigned (i.e., mapped) to "code points" (i.e., U+0000 to U+10FFFF) and character encodings which encode scalar values (all code points except the surrogate ranges, therefore including all characters).
Re:UTF-8 – "The most elegant hack" by duaneb on Saturday September 28, 2013 @10:18PM (#6463848)
Characters are not fixed width, no, but codepoints are. The idea of "what exactly constitutes a character" gets pretty fuzzy in some cases anyway with utility codepoints.
Re:UTF-8 – "The most elegant hack" by jameshart on Saturday September 28, 2013 @11:32PM (#6464002)
Even Windows-format ASCII isn't fixed width - CR-LF is a two-byte 'codepoint'...
Re:UTF-8 – "The most elegant hack" by nitrogen on Sunday September 29, 2013 @04:12AM (#6464511)
You can think of it as two separate instructions, CR to move horizontally to the beginning of the line, and LF to move down by one line.
Re:UTF-8 – "The most elegant hack" by jameshart on Monday September 30, 2013 @09:40AM (#6469534)
You can think of it that way until you say, try to take the first n characters of a string and wind up with your string terminated with an orphaned CR without its accompanying LF.

Outside of ANSI terminals, teletypes and and dot matrix printers, they really don't work like that any more. If they did you'd be able to substitute LFCR for CRLF, or use multiple LFs with a single CR to leave multiline gaps - and multiple CRs would be redundant. Try sending a CSV file with LFCR line-endings, or HTTP headers separated by CRCRCRCRLF... see if that works everywhere reliably. Also, CRLF would be an acceptable substitute everywhere for UNIX LF line-endings, but clearly it's not. Fact is, CRLF is a two-byte magic number in a lot of places.

Re:UTF-8 – "The most elegant hack" by zorked on Sunday September 29, 2013 @06:46AM (#6464768)
Not really because a single CR or LF does not do a real CR or LF.
Re:UTF-8 – "The most elegant hack" by prewett on Sunday September 29, 2013 @10:05AM (#6465202)
It used to, back in The Day, when you had hard-copy consoles or even dot matrix printers. I think sending CR (but not LF) and then printing accents over top of the original line was a way of printing some European languages with just ASCII.
Re:UTF-8 – "The most elegant hack" by barrkel on Sunday September 29, 2013 @03:46PM (#6466254)

  $ man terminfo | grep cursor_down | head -1
         cursor_down                   cud1       do        down one line
  $ tput cud1 | hexdump -C | cut -f 1-20 | head -1
  00000000  0a                                                |.|
  $ man terminfo | grep carriage_return | head -1
         carriage_return               cr         cr        carriage return (P*)
  $ tput cr | hexdump -C | cut -f 1-20 | head -1
  00000000  0d                                                |.|
If you ever write a bash script that does more advanced things with the terminal, you'll get familiar with tput, termcap and terminfo pretty quickly.
Re:UTF-8 – "The most elegant hack" by kps on Sunday September 29, 2013 @10:01PM (#6467416)
It does in ASCII, and (consequently) in any ANSI-conforming terminal [emulator].