News for hackers. Stuff that matters.
Re:UTF-8 – "The most elegant hack" by lmm on Sunday September 29, 2013 @01:35PM (#6465853)
Standard ruby is not utf8-only and not likely to be any time soon, because converting text into unicode is lossy due to han unification. So like it or not, you're going to need to deal with non-unicode text for a while yet.
Re:UTF-8 – "The most elegant hack" by dlitz on Sunday September 29, 2013 @04:35PM (#6466418)
I disagree:

- Ruby isn't UTF8-only, but Rails is.

- Unless I'm building a site that caters to CJK languages where Han unification is unacceptable, I'm not going to need to deal with non-unicode text. In 4 years of Rails development, I never set $KCODE to anything except 'UTF8'.

- Any solution to the Han unification problem is almost certainly going to happen within Unicode, or at least in some Unicode private-use area that can be encoded using UTF-8.

- As a last resort, it's still easier to use something like "surrogateescape"/"UTF-8b" to pass arbitrary bytes through the system than it is to support multiple text encodings at every single place where text is handled.

Re:UTF-8 – "The most elegant hack" by asveikau on Sunday September 29, 2013 @02:00PM (#6465948)
Very interesting, I guess it's just a function of not knowing much about that part of the world but I'd never heard of this problem before. http://en.wikipedia.org/wiki/Han_unification

Reading that article it's a wonder they didn't come up with a mode-switching character, similar to what the article says about ISO/IEC 2022 or (cringe) Unicode bi-di chars.

Unicode is one of those things that naively sounds great and everyone talks about as solving every problem, but it ends up having lots of warts...

Re:UTF-8 – "The most elegant hack" by asveikau on Sunday September 29, 2013 @06:23PM (#6466788)
Ah OK, reading again I see that I Unicode does provide a way to disambiguate with combining characters.