News for hackers. Stuff that matters.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by salmonellaeater on Monday August 26, 2013 @02:09AM (#6275456)
What is the argument for utf-8 that doesn't also apply to utf-16, depending on what your common characters are? Applications that use Asian languages take a big hit from utf-8.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by Dylan16807 on Monday August 26, 2013 @02:50AM (#6275561)
>What is the argument for utf-8 that doesn't also apply to utf-16

The comment you're replying to says right there, "implementations are routinely broken because the corner cases are rarer" utf-16 misleads people into using fixed-width code where it won't work right.

>Applications that use Asian languages take a big hit from utf-8.

Not really. It's a moderate hit for pure text, but how often do you see bulk pure text, without any markup, and that can't be compressed?

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by salmonellaeater on Monday August 26, 2013 @04:56AM (#6275900)
> utf-16 misleads people into using fixed-width code where it won't work right

This is exactly my point: lots of code also treats utf-8 as if it were ASCII.

> how often do you see bulk pure text, without any markup, and that can't be compressed?

That's a good argument. Thanks :)

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by stormbrew on Monday August 26, 2013 @01:01PM (#6278257)
> This is exactly my point: lots of code also treats utf-8 as if it were ASCII.

Two things to this:

- In many cases this is an entirely non-destructive mistake to make. This is an advantage of utf-8, that intermediaries that only care about ascii characters (even in the presence of extended characters) can work with it.

- It will become readily apparent that you've got a problem on the first attempt to port your app to any language with a non-ascii script. With UTF-16 it only becomes apparent when you go beyond the BMP.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by querulous on Monday August 26, 2013 @02:16AM (#6275472)
there are no surrogates in utf-8. that alone is reason to use it over utf-16
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by millstone on Monday August 26, 2013 @04:11AM (#6275767)
I don't get it. Surrogates in UTF-16 are when you have two code units that map to a single code point. In UTF-8, you have that same issue, except it may be two, three, or four code units. So how is UTF-8 better in this respect?