News for hackers. Stuff that matters.
Re:UTF-8 – "The most elegant hack" by mojuba on Saturday September 28, 2013 @06:31PM (#6463376)
We should be grateful to UTF-8 for saving us from fixed multibyte Internet (so persistently pushed by MS, IBM and others), which would have made little sense in this predominantly 8-bit world. I remember someone saying at the time: the future is 16-bit modems and floppy disks anyway, so why not switch to Unicode now? Somehow that sounded absurd to me.

Anyway, 20 years later, hardware is still mostly 8-bit, and basically nobody cares about Unicode apart from font designers and the Unicode Consortium.

(On a side note, UTF-8 as a hack is a distant relative to Huffman encoding, itself a beautiful thing.)

Re:UTF-8 – "The most elegant hack" by Stratoscope on Saturday September 28, 2013 @06:47PM (#6463407)
"16 bits should be enough for anybody." --Anonymous

The funny thing is that 16 bits wasn't enough.

We thought it would be, back then. Just use 16 bits for each character and we can return to the joy and simplicity of a fixed width encoding.

That didn't work out. UCS-2 was a fixed width 16-bit encoding, but it it had to be extended into the variable width UTF-16 to get more code points. You get the bloat of 16-bit data, incompatibility with ASCII, and variable width characters too.

You have to go all the way to UTF-32 to get back to a fixed width encoding.

So UTF-8 really was a smart move. Bite the bullet on the variable width, but make it so you can always find the beginning of the next character even if you start in the middle of one, and make it backward compatible with ASCII too. Genius.

https://en.wikipedia.org/wiki/UTF-8

https://en.wikipedia.org/wiki/UTF-16

https://en.wikipedia.org/wiki/UTF-32

Re:UTF-8 – "The most elegant hack" by sopooneo on Saturday September 28, 2013 @07:12PM (#6463450)
Please correct me if I'm wrong, but you don't really get fixed width in anyway do you? Because unicode itself is not fixed width. There are situations where one character on the screen is represented by two consecutive unicode code points.
Re:UTF-8 – "The most elegant hack" by plorkyeran on Saturday September 28, 2013 @07:46PM (#6463503)
Or a lot more than two, or even multiple characters for a single code point.

UTF-32 appears to simplify things a lot, but in practice O(1) indexing of code points and simple iteration over code points is not actually all that useful, because code points are not characters.

Re:UTF-8 – "The most elegant hack" by ygra on Sunday September 29, 2013 @04:27AM (#6464539)
There are multiple factors at play here and it ultimately comes down to what you need:

If you to access code points, then UTF-32 gives you a nice fixed-width (albeit quite large) encoding to work with. Some applications or libraries may use UTF-32 internally to speed things up.

If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points: e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾ (this probably doesn't even render properly).

Sadly, most APIs use either UTF-8 or UTF-16 and consider code units the most important entity, not code points. And that's inexcusable and frankly a mess, because I can count the number of times I needed to access a code unit instead of a code point on zero hands.

Re:UTF-8 – "The most elegant hack" by tripzilch on Sunday September 29, 2013 @06:42AM (#6464761)
> If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points:

> e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾

As how many characters am I supposed to perceive this, anyway? Some of those tiny squiggles are in fact letters (I mean they look like letters, I see an e,m,u,o,i and r in there). What if you'd make a crossword this way?

Re:UTF-8 – "The most elegant hack" by dbaupp on Sunday September 29, 2013 @07:34AM (#6464861)
It says "example" with a pile of combining characters (77 to be precise): http://i.imgur.com/acyHBkR.png
Re:UTF-8 – "The most elegant hack" by thaumasiotes on Sunday September 29, 2013 @09:16AM (#6465089)
I've experienced crosswords where e.g. the word going across uses a diacritic on a particular letter ("élan") and the word going down doesn't. It drives me crazy.
Re:UTF-8 – "The most elegant hack" by riffraff on Sunday September 29, 2013 @01:55PM (#6465932)
FYI, that is the way crosswords work in italian, accents are considered irrelevant.
Re:UTF-8 – "The most elegant hack" by iv_08 on Sunday September 29, 2013 @06:04AM (#6464705)
For index based operations code units are more useful than code points.

a = s.indexOf("e̬̱ͤ̈́̿̂̅́͟")

b = s.indexOf("m̛̮̩͙ͦ͗͆̄̋́̄")

s[a:b]

This operation is fast with code unit indices and slow with code point indices.

Re:UTF-8 – "The most elegant hack" by Dylan16807 on Sunday September 29, 2013 @01:39PM (#6465865)
If you use stream-safe normalization rules, then you have a worst-case limit of 30 combining code points in a row. You could even make an array of 128 byte single-character-strings if you wanted.
Re:UTF-8 – "The most elegant hack" by Flimm on Sunday September 29, 2013 @05:10AM (#6464610)
Please define what you mean by code point and code unit.
Re:UTF-8 – "The most elegant hack" by ygra on Sunday September 29, 2013 @07:47AM (#6464890)
I mean the same thing that Unicode uses these terms for.

Code Unit. The minimal bit combination that can represent a unit of encoded text for processing or interchange. The Unicode Standard uses 8-bit code units in the UTF-8 encoding form, 16-bit code units in the UTF-16 encoding form, and 32-bit code units in the UTF-32 encoding form.

Code Point. (1) Any value in the Unicode codespace; that is, the range of integers from 0 to 10FFFF16. (See definition D10 in Section 3.4, Characters and Encoding.) Not all code points are assigned to encoded characters. See code point type. (2) A value, or position, for a character, in any coded character set.

From http://www.unicode.org/glossary/

Re:UTF-8 – "The most elegant hack" by runn1ng on Saturday September 28, 2013 @07:50PM (#6463525)
UTF8 != unicode.

Unicode is a standard, that maps the giant number of letters to unique numbers.

UTF8 (and UTF16, UTF32) are various encodings of that. Unicode itself is not an encoding; unicode just says "letter 'č' is 269"; UTF-8 says "We will represent it as two bytes, first is C4, second is 8D".

UTF8 is indeed variable width; UTF16 and UTF32 are, AFAIK, fixed width. Both use unicode standard.

Re:UTF-8 – "The most elegant hack" by Stratoscope on Saturday September 28, 2013 @08:15PM (#6463592)
Even if we're talking about code points and not characters, UTF-16 is variable width.

UTF-8 is variable width using 8, 16, 24, or 32 bits per code point.

UCS-2 is fixed width using 16 bits per code point.

UTF-16 is variable width using 16 or 32 bits per code point. It's an extension of the fixed width UCS-2 format: every valid 16-bit UCS-2 code point is also a valid UTF-16 code point, but UTF-16 also includes 32-bit code points.

UTF-32 is fixed width using 32 bits per code point.

Re:UTF-8 – "The most elegant hack" by alister on Saturday September 28, 2013 @10:28PM (#6463870)
UTF-8 is variable width using 8, 16, 24, or 32 bits per code point.

I wondered why you didn't write that "UTF-8 is variable width using 8, 16, 24, 32, 40, or 48 bits per code point", because that's how Prosser/Thompson/Pike's UTF-8 idea was proposed, as shown in the "most elegant hack" article.

And here's the answer from Wikipedia: In November 2003 UTF-8 was restricted by RFC 3629 to end at U+10FFFF, in order to match the constraints of the UTF-16 character encoding. This removed all 5- and 6-byte sequences, and about half of the 4-byte sequences.

Even though they eliminated the 5th and 6th bytes, UTF-8 can still represent all of Unicode's 1,114,112 code points.

I'm mentioning this because others probably are wondering the same thing.

Re:UTF-8 – "The most elegant hack" by option_greek on Sunday September 29, 2013 @03:44AM (#6464455)
How sad. One defective format limits another great format.
Re:UTF-8 – "The most elegant hack" by ygra on Sunday September 29, 2013 @04:21AM (#6464527)
We don't see the code point space run out anytime soon this time, even with questionable additions like Emoji. It took us 20 years to get to the current state and character encoding rate has slowed down significantly since the early days (mostly because most things are covered already).
Re:UTF-8 – "The most elegant hack" by btilly on Saturday September 28, 2013 @11:14PM (#6463954)
UTF8 is indeed variable width; UTF16 and UTF32 are, AFAIK, fixed width. Both use unicode standard.

Your knowledge is wrong.

As of 2012, there are 110,182 assigned Unicode characters, and 137,468 more which have been reserved for private use.

With 16 bits there are only 65,536 possible values, and so UTF-16 has to be variable width. (Though it was originally intended to be fixed width. A misconception unfortunately embedded in the design of languages from the time such as Java.)

Re:UTF-8 – "The most elegant hack" by pjscott on Saturday September 28, 2013 @07:57PM (#6463538)
What the comment you're replying to is saying that glyphs may be represented by more than one code point. So sure, UTF32 has a fixed width for each code point, but it's not quite so clear how useful that is.
Re:UTF-8 – "The most elegant hack" by runn1ng on Saturday September 28, 2013 @08:01PM (#6463552)
Oh, in that case I misunderstood parent. Sorry about that.
Re:UTF-8 – "The most elegant hack" by gsnedders on Saturday September 28, 2013 @09:44PM (#6463771)
The Unicode standard nowadays defines UTF-8, UTF-16, and UTF-32. It speaks of "characters" which are assigned (i.e., mapped) to "code points" (i.e., U+0000 to U+10FFFF) and character encodings which encode scalar values (all code points except the surrogate ranges, therefore including all characters).
Re:UTF-8 – "The most elegant hack" by duaneb on Saturday September 28, 2013 @10:18PM (#6463848)
Characters are not fixed width, no, but codepoints are. The idea of "what exactly constitutes a character" gets pretty fuzzy in some cases anyway with utility codepoints.
Re:UTF-8 – "The most elegant hack" by jameshart on Saturday September 28, 2013 @11:32PM (#6464002)
Even Windows-format ASCII isn't fixed width - CR-LF is a two-byte 'codepoint'...
Re:UTF-8 – "The most elegant hack" by nitrogen on Sunday September 29, 2013 @04:12AM (#6464511)
You can think of it as two separate instructions, CR to move horizontally to the beginning of the line, and LF to move down by one line.
Re:UTF-8 – "The most elegant hack" by jameshart on Monday September 30, 2013 @09:40AM (#6469534)
You can think of it that way until you say, try to take the first n characters of a string and wind up with your string terminated with an orphaned CR without its accompanying LF.

Outside of ANSI terminals, teletypes and and dot matrix printers, they really don't work like that any more. If they did you'd be able to substitute LFCR for CRLF, or use multiple LFs with a single CR to leave multiline gaps - and multiple CRs would be redundant. Try sending a CSV file with LFCR line-endings, or HTTP headers separated by CRCRCRCRLF... see if that works everywhere reliably. Also, CRLF would be an acceptable substitute everywhere for UNIX LF line-endings, but clearly it's not. Fact is, CRLF is a two-byte magic number in a lot of places.

Re:UTF-8 – "The most elegant hack" by zorked on Sunday September 29, 2013 @06:46AM (#6464768)
Not really because a single CR or LF does not do a real CR or LF.
Re:UTF-8 – "The most elegant hack" by prewett on Sunday September 29, 2013 @10:05AM (#6465202)
It used to, back in The Day, when you had hard-copy consoles or even dot matrix printers. I think sending CR (but not LF) and then printing accents over top of the original line was a way of printing some European languages with just ASCII.
Re:UTF-8 – "The most elegant hack" by barrkel on Sunday September 29, 2013 @03:46PM (#6466254)

  $ man terminfo | grep cursor_down | head -1
         cursor_down                   cud1       do        down one line
  $ tput cud1 | hexdump -C | cut -f 1-20 | head -1
  00000000  0a                                                |.|
  $ man terminfo | grep carriage_return | head -1
         carriage_return               cr         cr        carriage return (P*)
  $ tput cr | hexdump -C | cut -f 1-20 | head -1
  00000000  0d                                                |.|
If you ever write a bash script that does more advanced things with the terminal, you'll get familiar with tput, termcap and terminfo pretty quickly.
Re:UTF-8 – "The most elegant hack" by kps on Sunday September 29, 2013 @10:01PM (#6467416)
It does in ASCII, and (consequently) in any ANSI-conforming terminal [emulator].
Re:UTF-8 – "The most elegant hack" by evincarofautumn on Saturday September 28, 2013 @08:00PM (#6463549)
> You have to go all the way to UTF-32 to get back to a fixed width encoding.

UTF-21, actually, as there are no code points above 0x10FFFF. Powers of two have other nice properties, though.

Re:UTF-8 – "The most elegant hack" by ygra on Monday September 30, 2013 @01:30AM (#6467974)
There is no UTF-21. The number after UTF signifies the size of the code unit, which in UTF-32 is 32 bits wide. That code points are only defined up to U+10FFFF is irrelevant for that; the code unit is still 32 bits.
Re:UTF-8 – "The most elegant hack" by evincarofautumn on Monday September 30, 2013 @02:52AM (#6468162)
Yeah, I was joking. UTF-32 is a waste of 11 perfectly good bits[1], so you could just as well write each code point as 21 bits (0o0000000–0o7777777), and call it UTF-21. 24 bits (3 × 0x00–0x7F) would be fine too, but 21 is the smallest possible fixed-width Unicode encoding.

[1]: https://github.com/evincarofautumn/protodata

Re:UTF-8 – "The most elegant hack" by asveikau on Saturday September 28, 2013 @07:15PM (#6463457)
> so persistently pushed by MS

People still flame how MS uses 16-bit characters, which IMO is a bit of an unfair criticism. (Full disclosure: I used to work for them. But I was never a koolaid drinker.) I don't think they have any religious aversion to UTF-8, it's just history. By adopting Unicode as early as they did, they made this decision before UTF-8 existed and they stuck with it.

Probably the bigger crime is not switching to UTF-8 as the default "ANSI"/"multi-byte" codepage (to use the Windows terms). This means C programmers who are not Windows experts often end up writing non-Unicode-safe software because they expect every string to be char*.

(Also as has been mentioned UTF-16 is not fixed. And even in UTF-32 there are instances where a single glyph takes multiple code points - the decomposed accent marks are the one example I know, possibly there are others?)

Re:UTF-8 – "The most elegant hack" by dlitz on Saturday September 28, 2013 @10:00PM (#6463802)
IMHO it's completely fair to criticize others for the externalities that they create. UTF-16 is a design flaw, and Microsoft repeatedly decides to release a new version of Windows that fails to meaningfully address that flaw. To date, MS hasn't communicated a vision for a future of Windows where nobody has to deal with UTF-16, we can only conclude that their vision is that for the next 100 years, the rest of us will still be paying to support their use of UTF-16.

This isn't just about UTF-16 either. Think about how much IE6 has cost everyone who's made a website in the last 10 years. Microsoft's penchant for building and marketing high-friction platforms is a huge drain on innovation (and a big barrier-to-entry for people who want to learn programming), and the company deserves an enormous amount of criticism for it.

Re:UTF-8 – "The most elegant hack" by asveikau on Saturday September 28, 2013 @11:30PM (#6463998)
Can you explain how utf-16 is a design flaw? This sounds totally kooky to me. Are you thinking of the more limited UCS-2? Utf-16 represents the same set of chars as utf-8. It's true that lots of errors can occur when people assume 1 wchar = 1 codepoint = 1 glyph but utf-8 has similar complexity and I've seen plenty of people screw it up too. It sounds much more to me like you are saying that "not working how I am used to" is the same as "design flaw".

Edit: put another way, the NT kernel since its inception represented and continues to represent all strings as 16 bit integers, saying that they need to "meaningfully address this" is like saying Linux should migrate away from 8-bit strings; there is no reason to do it. A lossless conversion exists and I fail to see it as a big deal, it's just a historical thing because NT's initial development predates UTF-8.

Re:UTF-8 – "The most elegant hack" by dlitz on Sunday September 29, 2013 @05:52AM (#6464687)
Imagine a world where we don't have to convert between multiple charsets, or even think about them, beyond "text" vs "binary". That world is perfectly achievable, it's basically already happened outside the Microsoft ecosystem.

OSX and Linux default to using UTF-8 everywhere, Ruby on Rails is UTF-8 only, the assumed source encoding in Python 3 is UTF-8, URI percent-encoding is done exclusively using UTF-8, HTML5 defaults to UTF-8... The list goes on and on. "UTF-8" is becoming synonymous with "text".

In an all-UTF-8 world, if you want to build a URI parser, or a CSV parser, or an HTML parser, or really anything that does any kind of text processing (except rendering), you can just assume ASCII and everything will work as long as you're 8-bit clean. Even non-US codepages are all more-or-less supersets of ASCII. The only major exceptions are EBCDIC and UTF-16.

Because of UTF-16, we have to build charset conversion/negotiation into every single interface boundary, where we could otherwise just be 8-bit clean except in places where text actually has to be rendered.

It's completely unnecessary friction that undermines one of the big motivations for creating Unicode in the first place: coming up with a single standard for text representation, so that we don't have to deal with the mess of handling multiple charsets everywhere.

None of these decisions is a big deal in isolation, but it's a death by 1000 cuts. When you have mountains of unnecessary complexity, it only serves to make programming inaccessible to ordinary people, which is how we get awful policies like SOPA, software patents, and the like.

Re:UTF-8 – "The most elegant hack" by asveikau on Sunday September 29, 2013 @07:50AM (#6464897)
So much hand-wavy text in there...

There's no inherent, fundamental reason in the universe other than momentum that says you've got to use an 8-bit encoding. You could serve to be a bit more honest about how arbitrary that call really is and where it comes from. In fact many higher level languages even outside the MSFT ecosystem are happily using UTF-16 everywhere natively without problems, they just do a small conversion step at a syscall to run on your beloved Unix. AFAIK the JVM is working this way for example. This "imagine a world" game can just as easily go the other way: "imagine a world in which everything is UTF-16 ...". I think it's disingenuous to say that it matters one way or the other, that your favorite is better and all others, even if they are a 1-to-1 mapping with your favorite, constitute a "design flaw". Because information theory does not care; it is just a different encoding for the same damn thing. In truth it makes little difference as long as you are consistent, and you are whining simply because not everybody picked the same thing as your favorite.

Your claim that UTF-16 is a "design flaw" has not been validated, just that you don't like encodings ever to change and UTF-16 isn't your favorite. It's very hard to consider that anything other than whining. I thought we software types are supposed to be big on abstractions and coming up with clever ways of managing complexity? It seems rather rigid to say you'll only ever deal with one text encoding.

Lastly even in 2013 this is a total lie:

> OSX and Linux default to using UTF-8 everywhere,

If that's true then how come on virtually every Unix-like system I've set up for the last ~15 years one of the first things I've had to do is edit ~/.profile to futz around with LC_CTYPE or whatever to ask for UTF-8? I am pretty sure every Unix-like system I have set up gave me Latin1 by default, even quite recently.

I think you are underestimating the extent to which UTF-8 is a crude hack designed to avoid rewriting ancient C programs that did very much the wrong approach to localization. There was a time before UTF-8 existed and became popular when it was a fairly common viewpoint that proper Unicode support involved making a clean break with the old char type. You are right to say that UTF-8 "won" in most places but the fact that the NT kernel or the JVM use 16-bit chars reflects that prior history. I think a more mature attitude would be to accept this, that it came from a time and a place and is a different way of working, rather than call it "wrong".

Re:UTF-8 – "The most elegant hack" by dlitz on Sunday September 29, 2013 @04:00PM (#6466300)
> There's no inherent, fundamental reason in the universe other than momentum that says you've got to use an 8-bit encoding.

"Other than momentum"? That's a double-standard: Momentum is the only reason why UTF-16 is still relevant today. Ignoring momentum, UTF-8 still has a bunch of advantages over UTF-16, namely that endianness isn't an issue, it's self-synchronizing over byte-oriented communication channels, and it's more likely to be implemented correctly (bugs related to variable-length encoding are much less likely to get shipped to users, because they start to occur as soon as you step outside the ASCII range, rather than only once you get outside the BMP). What advantages does UTF-16 have, ignoring momentum?

The comparison between UTF-8 and UTF-16 is adequately addressed here: http://www.utf8everywhere.org/

I don't want to debate abstract philosophy with you, anyway. Momentum may be the reason, but there's no plausible way that UTF-16 is ever going to replace octet-oriented text. The idea that UTF-8 and UTF-16 are equivalent in practice is a complete fantasy, and I'm arguing that we should pick one, rather than always having to manage multiple encodings.

> I thought we software types are supposed to be big on abstractions and coming up with clever ways of managing complexity?

The best way to manage complexity is usually to adopt practices that tend to eliminate it over time, rather than adding more complexity in an attempt to hide previous complexity. It doesn't matter how "clever" that sounds, but it's generally accepted that it takes more skill and effort to make things simpler than it does to make them more complex.

> It seems rather rigid to say you'll only ever deal with one text encoding.

It seems rather rigid to say you'll only ever deal with two's-complement signed integer encoding. It seems rather rigid to say you'll only ever deal with IEEE 754 floating-point arithmetic. It seems rather rigid to say you'll only ever deal with 8-bit bytes. It seems rather rigid to say you'll only ever deal with big-endian encoding on the network. It seems rather rigid to say you'll only ever deal with little-endian encoding in CPUs. It seems rather rigid to say you'll only ever deal with TCP/IP.

Why not eventually only ever deal with one text encoding? There's no inherent value in paying engineers to spend their time thinking about multiple text encodings, everywhere, forever.

Remember that we're talking about the primary interfaces for exchanging text between software components. Sure, there are occasions where someone needs to deal with other representations, but the smart thing to do is to pick a standard representation and move the conversion/negotiation stuff into libraries that only need to be used by the people who need them. This allows the rest of us to quit paying for the unnecessary complexity, and incentivizes people to move toward the standard representation if their need for backward compatibility doesn't outweigh the cost of actually maintaining it.

> I am pretty sure every Unix-like system I have set up gave me Latin1 by default, even quite recently.

Ubuntu, Debian, and Fedora all default to UTF-8, and have for several years now. You're going to have to name names, or I'm going to assume that you don't know what you're talking about.

> I think you are underestimating the extent to which UTF-8 is a crude hack designed to avoid rewriting ancient C programs that did very much the wrong approach to localization.

Really? How would you have done it so that UTF-16 wouldn't have broken your program? Encode all text strings as length-prefixed binary data, even inside text files? It's ironic that you say it's immature to call UTF-16 "wrong", but you've basically just claimed that structured text in general is "wrong".

Let's not forget that nearly every important pre-Unicode text representation was at least partly compatible with ASCII: ISO-8859-* & EUC-CN were explicitly ASCII supersets, Shift-JIS & Big5 aren't but still preserve 0x00-0x3F, and even EBCDIC, NUL is still NUL. Absent an actual spec, it was no less reasonable to expect that an international text encoding would be ASCII-compatible than it was to expect that such a spec would break compatibility with erverything. Trying to anticipate the latter would rightly have been called out as overengineering, anyway.

In that environment, writing a CSV or HTML parser that handles a minimal number of special characters and is otherwise 8-bit clean is exactly the right approach to localization.

Also, "ancient C programs"? Seriously? Are you really saying that C and its calling convention were/are irrelevant?

> There was a time before UTF-8 existed and became popular when it was a fairly common viewpoint that proper Unicode support involved making a clean break with the old char type.

Sure, and it was a fairly common viewpoint that OSI was the right approach, and that MD5 was collision-resistant. Then, we learned that all of these viewpoints turned out to be wrong, and they were supplanted by better ideas. UTF-8 became popular because it worked better than trying to "redefine ALL the chars".

> You are right to say that UTF-8 "won" in most places but the fact that the NT kernel or the JVM use 16-bit chars reflects that prior history.

The JVM isn't comparable, because its internal representation is invisible to Java developers. I can write Java code without ever thinking about UTF-16, and a new version of the JVM could come out that changed the internal representation, and it wouldn't affect me. Python used a similar internal representation, and recently did change its internal representation. Most Python developers won't even notice.

If NT used UTF-16 internally, but provided a UTF-8 system call interface, I wouldn't care. The problem is that, in 2013, people writing brand new code on Windows still have to concern themselves with UTF-16 vs ANSI vs UTF-8. This is a pattern of behavior at Microsoft, and that's what I'm criticizing.

> I think a more mature attitude would be to accept this, that it came from a time and a place and is a different way of working, rather than call it "wrong".

Look, I understand that mistakes will be made. My criticism isn't that the mistakes are made in the first place, but that Microsoft doesn't appear to have any plan to ever rectify them. The result is a consistent increase in friction over time, the cost of which is mostly paid for by entities other than Microsoft.

One could argue that Microsoft's failure to manage complexity in this way is one of the reasons why Linux is eating their lunch in the server market. Anecdotally, it's just way easier to build stuff on top of Linux, because there's a culture of eliminating old cruft---or at least moving it around so that only the people who want it end up paying for it.

As for your ad hominem arguments about "attitude" and "maturity", I could do without them, thanks. They contribute nothing to the conversation, and only serve to undermine your credibility. Knock it off.

Re:UTF-8 – "The most elegant hack" by asveikau on Sunday September 29, 2013 @04:20PM (#6466364)
> The idea that UTF-8 and UTF-16 are equivalent in practice is a complete fantasy,

Except that most obscure of details that they represent the same characters.

> Ubuntu, Debian, and Fedora all default to UTF-8, and have for several years now. You're going to have to name names,

In the room where I'm sitting now I have Debian, OpenBSD and Arch systems. All of these defaulted to Latin-1 when I installed them, in the current decade.

> I can write Java code without ever thinking about UTF-16

Patently false. A char in java is 16 bits. String.length() and String.charAt() use utf-16 units, meaning surrogate pairs are double-counted.

> As for your ad hominem arguments about "attitude" and "maturity", I could do without them, thanks.

I am sorry, sometimes I am blunt and overstated about these characterizations, but it really did seem like the shoe fits. You seem to have an impulsive defense of UTF-8 and an inability to see that there might be merits or tradeoffs in an alternative. Throughout all of this, I am not saying that UTF-8 is a bad encoding, I am just saying it's goofy to "attack" UTF-16 for being different.

Re:UTF-8 – "The most elegant hack" by lmm on Sunday September 29, 2013 @01:35PM (#6465853)
Standard ruby is not utf8-only and not likely to be any time soon, because converting text into unicode is lossy due to han unification. So like it or not, you're going to need to deal with non-unicode text for a while yet.
Re:UTF-8 – "The most elegant hack" by dlitz on Sunday September 29, 2013 @04:35PM (#6466418)
I disagree:

- Ruby isn't UTF8-only, but Rails is.

- Unless I'm building a site that caters to CJK languages where Han unification is unacceptable, I'm not going to need to deal with non-unicode text. In 4 years of Rails development, I never set $KCODE to anything except 'UTF8'.

- Any solution to the Han unification problem is almost certainly going to happen within Unicode, or at least in some Unicode private-use area that can be encoded using UTF-8.

- As a last resort, it's still easier to use something like "surrogateescape"/"UTF-8b" to pass arbitrary bytes through the system than it is to support multiple text encodings at every single place where text is handled.

Re:UTF-8 – "The most elegant hack" by asveikau on Sunday September 29, 2013 @02:00PM (#6465948)
Very interesting, I guess it's just a function of not knowing much about that part of the world but I'd never heard of this problem before. http://en.wikipedia.org/wiki/Han_unification

Reading that article it's a wonder they didn't come up with a mode-switching character, similar to what the article says about ISO/IEC 2022 or (cringe) Unicode bi-di chars.

Unicode is one of those things that naively sounds great and everyone talks about as solving every problem, but it ends up having lots of warts...

Re:UTF-8 – "The most elegant hack" by asveikau on Sunday September 29, 2013 @06:23PM (#6466788)
Ah OK, reading again I see that I Unicode does provide a way to disambiguate with combining characters.
Re:UTF-8 – "The most elegant hack" by malkia on Sunday September 29, 2013 @01:13AM (#6464201)
You need two versions of functions. You need two modes for lots of text programs to use. It doubles the space for things like storing identifiers for programming languages, which normally would use up to 7 bits (for big percentage of the languages).

UTF-8 can work with all 7-bit ASCII characters, and that's what it's great about it.

Re:UTF-8 – "The most elegant hack" by asveikau on Sunday September 29, 2013 @01:22AM (#6464219)
> You need two versions of functions.

For Windows this is entirely a compatibility thing. One could imagine a world in which that was not strictly necessary. A good best practice for a Win32 app is to always use 16-bit strings when calling system APIs and pretend the "ANSI" versions don't exist; I would not recommend anything else.

In NT the 8-bit versions generally do nothing but convert to 16-bit and call the "real" function. In recent versions (I think Win7 was the first to do this), AFAIK the 8-bit shims typically exist in another module, so if you don't use them they don't get loaded. In NT on ARM the 8-bit shims are not even there.

> It doubles the space for things like storing identifiers for programming languages

Which is why GetProcAddress() still takes an 8-bit string. Just because the kernel (and hence the syscall interface) uses 16-bit everywhere doesn't mean you can't use 8-bit strings in your own process, or that you can't selectively pick what makes sense for your use.

(By the way, none of what I'm saying is a criticism of UTF-8. I think it's a very clever encoding.)

Re:UTF-8 – "The most elegant hack" by option_greek on Sunday September 29, 2013 @03:52AM (#6464475)
If they were using UTF-8, wouldn't just one version of function suffice ? That is, instead of using a shim, just determine which version to call based on the encoding.
Re:UTF-8 – "The most elegant hack" by nitrogen on Sunday September 29, 2013 @04:16AM (#6464518)
There would be no need for the 16-bit versions if they had used UTF-8. The same function signatures would work for ASCII, ISO-8859, or UTF-8.
Re:UTF-8 – "The most elegant hack" by asveikau on Sunday September 29, 2013 @08:01AM (#6464926)
NT development began before UTF-8 existed. Surely you're not suggesting that the authors should have used an encoding that did not yet exist?
Re:UTF-8 – "The most elegant hack" by dmytrish on Sunday September 29, 2013 @02:55PM (#6466121)
Another example of UTF-16 brokenness: it is byte-order dependent, that is, programmer must care about big-endian vs little-endian encoding of 16-bit numbers. This caused another hack: using non-text BOM (Byte-Order Mark) symbols to denote what endianness is used.

I think it's very ugly and may be ok for Windows-x86 worlds, but not acceptable for variety of platforms connected to the Internet.

UTF-8 avoids endianness problems completely.

Re:UTF-8 – "The most elegant hack" by asveikau on Sunday September 29, 2013 @05:46PM (#6466656)
A legit complaint. Though I'll say, in early Unicode before UTF-8 existed, this was probably seen as less of a problem than using the older non-Unicode charsets, and probably rightly so.

Also, if you keep a clean separation between serialization and in-memory representation, the BOM hack is not such a big deal. Serialize code can write a BOM, and de-serialize can do the char swapping and omit the BOM in RAM, then in-memory it's always host byte order and without BOM. Or you can just have the on-disk/over-the-wire format be UTF-8, which is common today on platforms where UTF-16 is used in RAM. (I'll repeat a point that seems to be lost on a lot of people in this thread, that Microsoft adopted 16-bit chars for NT before UTF-8 existed, so that was not an option.)

Re:UTF-8 – "The most elegant hack" by erichurkman on Saturday September 28, 2013 @07:26PM (#6463469)
I still remember the day that the "View > Encoding" menu of browsers was absolutely critical in browsing the web if you browsed enough sites that were outside of your normal locale (your default codepage in Windows). The menu is still there, at least in Firefox & Chrome, but I haven't used it in a few years, as most things Just Work™ for modern websites, largely thanks to UTF-8.
Re:UTF-8 – "The most elegant hack" by gsnedders on Saturday September 28, 2013 @09:46PM (#6463777)
The eventual browser default is still (sadly) browser locale dependent, so the menu exists for sites designed for a different locale. I and some others still hope one day we can get rid of that (likely replaced with per domain, primarily TLDs, defaults).
Re:UTF-8 – "The most elegant hack" by deathanatos on Saturday September 28, 2013 @11:07PM (#6463942)
In IE, you mean? A standards-compliant browser shouldn't need to guess at the encoding. It can be found in the <meta> tag or (if that wasn't supplied or we're not dealing with HTML) in the Content-Type header. If it isn't found there, I believe HTML says, "it's ISO-8859-1 (latin1)".

That said, certain browsers have been known to guess.

Re:UTF-8 – "The most elegant hack" by gsnedders on Sunday September 29, 2013 @11:11AM (#6465362)
IE, Firefox, Chrome, Opera, Safari all have locale-dependent defaults for HTML. The process used is essentially: user-override, BOM, higher-level metadata (e.g., Content-Type in HTTP), meta pre-parsing, and then a locale-dependent default (Windows-1252 in most locales). Anything labelled "ISO-8859-1" is actually treated as Windows-1252 in browsers (they differ only in ISO-8859-1's C1 range, so Windows-1252 is a graphical superset).

http://www.whatwg.org/specs/web-apps/current-work/multipage/... has the detail of what all browsers implement nowadays.

Re:UTF-8 – "The most elegant hack" by Patrick_Devine on Saturday September 28, 2013 @08:22PM (#6463607)
I love that UTF-8 kind of followed in the tradition already set by the guys who came up with ASCII. ASCII itself is brilliant in that they aligned characters to be easy to test. Take upper and lower case 'A' which are 0x41 and 0x61 respectively:

0x41 0100 0001 0x61 0110 0001

So if you want to test whether a character is uppercase or not, you can just & 0x20.

Another one is how the number digits got lined up, which start at 0x30 for 0 to 0x39 for 9, which look like:

0x30 0011 0000 0x31 0011 0001 0x32 0011 0010 etc.

You can just read off the first nibble to convert from ASCII to an int which is pretty cool.

Re:UTF-8 – "The most elegant hack" by contingencies on Saturday September 28, 2013 @07:43PM (#6463498)
Anyway, 20 years later, hardware is still mostly 8-bit, and basically nobody cares about Unicode apart from font designers and the Unicode Consortium.

Err, failure to parse sarcasm?

Most people on most networks use it daily. On mobile networks, it's used for tens of thousands of messages per hour as SMS messages in almost every country with non-Romanized scripts uses UCS-2 encoding. I've had to write a raw SMS PDU generator for a whole bunch of human languages, including right to left ones like Farsi and Arabic, or Chinese, Japanese, Korean and Thai. Furthermore, Unicode is now used by everyone in China, having finally mostly migrated from GB encoding. That means "most of the internet uses it daily".

I guess your point may have been sarcastic, because the above is obvious to me. Either that, or you have a really bad case of a merry centrism.

Re:UTF-8 – "The most elegant hack" by ufmace on Saturday September 28, 2013 @09:31PM (#6463740)
I think he means "nobody cares about Unicode" in the sense that nobody cares about the technology itself. The guy in Japan probably loves that he can make a website in Japanese and have it look right on any system in the world. He probably loves even more that this just works without him having to know or care about encoding systems. Even most programmers using modern languages and frameworks can get away with not paying much attention to encodings and still probably have good support for international characters.
Re:UTF-8 – "The most elegant hack" by contingencies on Saturday September 28, 2013 @09:39PM (#6463758)
OK, thanks. So you've translated that "nobody cares" apparently means "everyone uses it and it works really well". And that's supposed to be a point? God, I must be getting old.

(Edit: I am not getting frustrated with you, rather my incapacity to grok the original post.)

Re:UTF-8 – "The most elegant hack" by jlgreco on Saturday September 28, 2013 @10:13PM (#6463839)
I don't know why you are getting frustrated with him, he just (correctly, as far as I can tell) interpreted the above comment for you.
Re:UTF-8 – "The most elegant hack" by mercurial on Sunday September 29, 2013 @10:39AM (#6465268)
> The guy in Japan probably loves that he can make a website in Japanese and have it look right on any system in the world.

That guy is probably using Shift-JS.

Re:UTF-8 – "The most elegant hack" by ttflee on Sunday September 29, 2013 @01:20AM (#6464218)
> Unicode is now used by everyone in China, having finally mostly migrated from GB encoding

I doubt it.

Actually Baidu and other major sites are still on GB2312/GB18030. There are practical reasons as most of the frequently used (3000+ characters) Chinese glyphs take two bytes in GB2312, but three in UTF-8. Many URIs contain escaped GB2312 instead of escaped UTF-8. For mobile(, not only 3G, but more significantly, the GPRS/EDGE/WAP) users, every bit counts.

Re:UTF-8 – "The most elegant hack" by LukeShu on Sunday September 29, 2013 @12:24AM (#6464119)
Uses it != cares about it
Re:UTF-8 – "The most elegant hack" by jrochkind1 on Sunday September 29, 2013 @12:16AM (#6464109)
Nobody cares about unicode?

I couldn't write web software that deals with non-ascii, and especially non-Latin, alphabets without paying attention to Unicode.

When I start getting into weird edge and corner issues -- I am continually amazed that Unicode works as well as it does. It turns out modeling alphabets is way more complicated than you might think. And Unicode gets an amazing amount right, making it practical to do things robustly correct.

Including not just the character set and encodings, but things like Unicode collation algorithms. A bunch of really smart people have worked through a bunch of different non-trivial problems of dealing with diverse alphabets, and provided solutions you can use that pretty much just work, bug-free.

Re:UTF-8 – "The most elegant hack" by statictype on Saturday September 28, 2013 @08:20PM (#6463603)
and basically nobody cares about Unicode apart from font designers and the Unicode Consortium.

If by nobody you mean, pretty much the entire world east of New York.

Re:UTF-8 – "The most elegant hack" by stephencanon on Saturday September 28, 2013 @08:28PM (#6463621)
Just the other day I was having a good laugh with the carpenters building my house here in New Hampshire at the expense of the rubes in Maryland who still only use ASCII.
Re:UTF-8 – "The most elegant hack" by mistercow on Sunday September 29, 2013 @05:11AM (#6464615)
These days, it wouldn't make much difference in terms of bandwidth, since everyone gzips everything on the server side.
Re:UTF-8 – "The most elegant hack" by ww520 on Saturday September 28, 2013 @07:47PM (#6463507)
And thanks to Java popularizing UTF-8. Java was the major language to support Unicode natively. The lack of Unicode support in C++ was disappointing.

(Yes, UTF-8 hack is nothing compared to Huffman coding. The Deflate compression algorithm using Huffman coding to encode the LZ77 output and the Huffman encoding table itself is another notch up on the hacking coolness scale.)

Re:UTF-8 – "The most elegant hack" by shadowmatter on Saturday September 28, 2013 @08:53PM (#6463665)
From http://docs.oracle.com/javase/7/docs/api/java/nio/charset/Ch...: "The native character encoding of the Java programming language is UTF-16."

That said, at least they adopted a Unicode encoding, and helped it establish mindshare.

Re:UTF-8 – "The most elegant hack" by sanderjd on Saturday September 28, 2013 @08:21PM (#6463606)
Is this right? Doesn't Java use UTF-16 internally?
Re:UTF-8 – "The most elegant hack" by rspeer on Sunday September 29, 2013 @02:35AM (#6464340)
And Java's first attempt to implement UTF-8, the one they called "UTF8", is not UTF-8. It's almost but not quite CESU-8. It will probably fuck up your astral characters (like emoji).

The real Java UTF-8 is, intuitively, named "AL32UTF8".

Re:UTF-8 – "The most elegant hack" by davidgerard on Sunday September 29, 2013 @04:45PM (#6466455)
And then there's the unremitting horror that MySQL calls UTF-8, properly known as WTF-8.
Re:UTF-8 – "The most elegant hack" by rspeer on Wednesday October 02, 2013 @03:45AM (#6481210)
I'd want to save the name "WTF-8" for that case where someone takes some text encoded as UTF-8, decodes it as if it's in a single-byte encoding, and then encodes the resulting mojibake as UTF-8 again.

Double UTF-8. WTF-8.