The flaws of thinking "Just use Unicode"

Re:The flaws of thinking "Just use Unicode" by ajross on Monday November 16, 2009 @12:04PM (#944732)

The whole premise of the link is flawed. It's an article about "localization is hard". Well, duh.

I don't see anything in it, at all, about character set and encoding choice. And in that realm, yes: you should just be using UTF-8. Not even "unicode" -- you want to be in UTF-8, period. Convert at the edges of your system if you absolutely must handle other encodings. But wherever you can, you should just be using UTF-8.

[ Parent | on HN ]

Re:The flaws of thinking "Just use Unicode" by lucumo on Monday November 16, 2009 @12:19PM (#944774)

The premise of the article wasn't that you shouldn't use Unicode. It was that you need to care about more than just Unicode. Thinking that the answer to localisation is to "just use Unicode" is a flaw in people's thinking that this article points out and corrects.

I'm curious as to why you believe UTF-8 is the only encoding you should use? I know of at least one person that says you should use UTF-16, because then it's immediately obvious when a certain piece of data is in the wrong encoding, whereas with UTF-8 you need to look for non-Latin characters (or accents) to distinguish it from, for example, ISO-8859-1. That line of thinking holds some value to me. Why do you believe otherwise?

[ Parent | on HN ]

Re:The flaws of thinking "Just use Unicode" by ajross on Monday November 16, 2009 @12:55PM (#944867)

UTF-16 can't be used with traditional tools. No grepping, no strings, no scanning quickly with a text editor. It locks you into whatever oddball toolchain you cooked up in development. Old code that passes the string to a strcat() will fail in mysterious ways. And your old libraries probably expect 1-byte characters anyway, which means you're going to be constantly converting

The inability to distinguish between UTF-8 and ASCII is a feature, not a bug. If you want to know if it's a valid string, just validate it. UTF-8 has extraordinarily strong validation properties, it's almost impossible for strings in other encodings to parse successfully.

[ Parent | on HN ]

Re:The flaws of thinking "Just use Unicode" by lucumo on Monday November 16, 2009 @02:25PM (#945076)

> Old code that passes the string to a strcat() will fail in mysterious ways. And your old libraries probably expect 1-byte characters anyway, which means you're going to be constantly converting

But they will have that same problem with UTF-8, right? Just as soon as you start using non-Latin characters? And if you aren't, there's no point in going beyond ASCII.

[ Parent | on HN ]

Re:The flaws of thinking "Just use Unicode" by ajross on Monday November 16, 2009 @05:15PM (#945428)

Uh, no, not at all. The ANSI strXXX() functions work just fine with UTF-8, that was kind of the point. Anything treating strings with nul termination or counted byte length works without change. Any work involving only ASCII substrings (e.g. parsing of computer-readable data) works without change. No non-ASCII character is represented using ASCII bytes in the string. Everything Just Works. Which is why you should just use it.

The sole exception are algorithms that truly need random access to characters within the string (e.g. "give me the 32nd character"), which can't interoperate with any variable-length encoding. But there are precious few of these. All anyone cares about is substrings.

[ Parent | on HN ]