Re:The flaws of thinking "Just use Unicode"
by lucumo on Monday November 16, 2009 @02:25PM
(#945076)
> Old code that passes the string to a strcat() will fail in mysterious ways. And your old libraries probably expect 1-byte characters anyway, which means you're going to be constantly converting
But they will have that same problem with UTF-8, right? Just as soon as you start using non-Latin characters? And if you aren't, there's no point in going beyond ASCII.
Re:The flaws of thinking "Just use Unicode"
by ajross on Monday November 16, 2009 @05:15PM
(#945428)
Uh, no, not at all. The ANSI strXXX() functions work just fine with UTF-8, that was kind of the point. Anything treating strings with nul termination or counted byte length works without change. Any work involving only ASCII substrings (e.g. parsing of computer-readable data) works without change. No non-ASCII character is represented using ASCII bytes in the string. Everything Just Works. Which is why you should just use it.
The sole exception are algorithms that truly need random access to characters within the string (e.g. "give me the 32nd character"), which can't interoperate with any variable-length encoding. But there are precious few of these. All anyone cares about is substrings.
The inability to distinguish between UTF-8 and ASCII is a feature, not a bug. If you want to know if it's a valid string, just validate it. UTF-8 has extraordinarily strong validation properties, it's almost impossible for strings in other encodings to parse successfully.