News for hackers. Stuff that matters.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by ghc on Sunday August 25, 2013 @11:57PM (#6275111)
Contrary to the editorialized title (which I'm sure will be changed soon), here's the best article on Unicode I've ever read: http://www.joelonsoftware.com/articles/Unicode.html
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by stormbrew on Monday August 26, 2013 @12:51AM (#6275260)
I feel like we're at a point now where articles that just try to 'demystify' unicode are almost teaching the controversy if they don't come out and actually say how you should deal with encodings in new apps.

It's about time we actually start pressing for the idea that utf-16 was a terrible idea and that utf-8 should be the dominant wire format for unicode, with ucs4 if you really need to have a linear representation.

Utf-16 is confusing, complicated, and implementations are routinely broken because the corner cases are rarer. I really hope we're not still stuck with it in 50 years.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by millstone on Monday August 26, 2013 @03:57AM (#6275730)
I agree about wire formats. However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those that use UTF-16. I think this is because with UTF-8, a lot of stuff just seems to work and so it's easy to ignore the issues, while UTF-16 forces you to come to grips with the realities of Unicode.

For example, consider a function to open a file by name. If your strings are UTF-8, you can just pass your null terminated buffer to fopen() or something and things will work fine for most files. But if your strings are internally UTF-16, you have to think about which encoding to use, and you research it, and you discover that holy crap, this stuff differs across OSes, and so we better take this problem seriously.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by stormbrew on Monday August 26, 2013 @08:41PM (#6280782)
> However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those that use UTF-16.

I'm really curious which languages drove you to this conclusion. Right now we're in a state where the most mature implementations of unicode in general are utf-16 because of the ucs2 accident, so it wouldn't exactly surprise me, but I'd still like to see the proof.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by plorkyeran on Monday August 26, 2013 @12:14PM (#6277951)
UTF-16 has the same issue with invalid things appearing to work due to surrogate pairs. A lot of UTF-16 based applications actually only support UCS-2, because you can assume one string index equals one character and get away with it if it never occurs to you to try to use stuff outside of the BMP.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by aerolite on Monday August 26, 2013 @03:33AM (#6275669)
More ethnocentrism. UTF-16 results in less bandwidth for any language that has non-ASCII characters.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by stormbrew on Monday August 26, 2013 @06:15PM (#6280257)
I think the Unicode group's handling of the round trip issues faced by Japanese and Chinese users with their own encodings due to Han Unification is a bigger ethnocentric problem by far. I'm not unaware of these issues.

But it's not like utf-16 was invented to solve that problem so much as it was an accident that in the early days of unicode 2 bytes was enough to encode all code points and a 2 byte encoding was appealing as a universal solution in a way that a 3 or 4 byte encoding never would be. Some group of people on the planet will have to suffer 3 or more byte encodings no matter what. And I'd be happy to make it mine if it meant we could go with the better technical solution, if I had the power to do that.

All that aside it's certainly not quite as clear cut as any language. Many languages have a mix of ascii and non-ascii characters and those fare better than they would with utf-16. And in the codepoints below 0x07ff (where utf-8 goes up to 3 bytes) there are character sets like greek, hebrew, arabic, and cyrillic. And as has been pointed out, ascii characters are often used in structural elements of documents and this factors into making it unlikely that the general case is as bad as the worst case (50% inflation) for most practical text. Particularly when you throw compression into the mix.

[edit] fixed some minor things.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by pedrocr on Monday August 26, 2013 @06:15AM (#6276115)
That's not true. For anything that uses the latin alphabet utf8 will be strictly better as the majority of characters will be in ASCII (1 byte) and the rest will just need an extra byte, equaling utf16.

Looking at the format definition[1] anything below U+0800 will fit in 2 bytes in UTF8. So the ethnocentrism starts at Samaritan[2]. The Japanese, Chinese and Indian scripts are probably the most important in that range.

[1] https://en.wikipedia.org/wiki/UTF-8#Description

[2] http://codepoints.net/basic_multilingual_plane

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by gsnedders on Monday August 26, 2013 @09:20AM (#6276817)
I can't find the research hsivonen (probably best known for validator.nu and Gecko's HTML parser) did into this a while ago, but he concluded that for the top 100 (IIRC) CJK sites, they were smaller as gzip'd UTF-8 than gzip'd UTF-16, because they all contained a sufficient quantity of HTML/CSS/JS that the gain from the shorter representation of them outweighed that of the text itself.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by salmonellaeater on Monday August 26, 2013 @02:09AM (#6275456)
What is the argument for utf-8 that doesn't also apply to utf-16, depending on what your common characters are? Applications that use Asian languages take a big hit from utf-8.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by Dylan16807 on Monday August 26, 2013 @02:50AM (#6275561)
>What is the argument for utf-8 that doesn't also apply to utf-16

The comment you're replying to says right there, "implementations are routinely broken because the corner cases are rarer" utf-16 misleads people into using fixed-width code where it won't work right.

>Applications that use Asian languages take a big hit from utf-8.

Not really. It's a moderate hit for pure text, but how often do you see bulk pure text, without any markup, and that can't be compressed?

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by salmonellaeater on Monday August 26, 2013 @04:56AM (#6275900)
> utf-16 misleads people into using fixed-width code where it won't work right

This is exactly my point: lots of code also treats utf-8 as if it were ASCII.

> how often do you see bulk pure text, without any markup, and that can't be compressed?

That's a good argument. Thanks :)

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by stormbrew on Monday August 26, 2013 @01:01PM (#6278257)
> This is exactly my point: lots of code also treats utf-8 as if it were ASCII.

Two things to this:

- In many cases this is an entirely non-destructive mistake to make. This is an advantage of utf-8, that intermediaries that only care about ascii characters (even in the presence of extended characters) can work with it.

- It will become readily apparent that you've got a problem on the first attempt to port your app to any language with a non-ascii script. With UTF-16 it only becomes apparent when you go beyond the BMP.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by querulous on Monday August 26, 2013 @02:16AM (#6275472)
there are no surrogates in utf-8. that alone is reason to use it over utf-16
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by millstone on Monday August 26, 2013 @04:11AM (#6275767)
I don't get it. Surrogates in UTF-16 are when you have two code units that map to a single code point. In UTF-8, you have that same issue, except it may be two, three, or four code units. So how is UTF-8 better in this respect?
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by SonOfLilit on Monday August 26, 2013 @04:12AM (#6275770)
You're probably looking for this:

http://www.utf8everywhere.org/

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by joel_perl_prog on Monday August 26, 2013 @12:09AM (#6275151)
Interesting talk by Nick Patch of Shutterstock about Unicode, why they use Perl, and what he calls best practices for working with encodings: http://www.youtube.com/watch?v=X2FQHUHjo8M
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by greenyoda on Sunday August 25, 2013 @11:40PM (#6275067)
HN guidelines ask you to retain the original title of the article, not to inject your own editorial opinion[1]. This article is a decent quick overview for someone who has never studied Unicode before, but a bit light on the details compared to what's available, say, in Wikipedia's articles on Unicode[2].

[1] http://ycombinator.com/newsguidelines.html

[2] https://en.wikipedia.org/wiki/Unicode

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by KayEss on Monday August 26, 2013 @03:27AM (#6275656)
>Other common myths include: Unicode can only support characters up to 65,536

Not really a myth. This was UCS2 and was the situation when a load of important early adopters started with Unicode. Winodws, Java, JavaScript all got burnt by this and ended up with UTF-16 as a result. Even Python 2.x on Linux is UTF-16 under the covers :(

>Unicode is just a standard way to map characters to magic numbers and there is no limit on the number of characters it can represent.

Unicode now limits itself to 21 bits of data. This is what allows the surrogate pair coding of UTF-16

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by cbr on Monday August 26, 2013 @07:19AM (#6276304)
At first there was no plain text. Then ASCII became standard enough and if you passed a file to someone they would be able to read it without you needing to tell them an encoding. Extensions of ASCII and other options spread, and there was no way to have plain text include characters beyond the 128 of ASCII. Over the past decade or so we've been consensing on utf8, however, and the probability that some random piece of non-ASCII text is utf8 has been getting sufficiently high.

There is plain text, and it is utf8.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by Argorak on Monday August 26, 2013 @03:11AM (#6275613)
By the way: A very interesting article for those interested in a bit of context around unicode and how it doesn't solve everything (especially from an asian perspective) is this one:

http://web.archive.org/web/20090627072117/http://www.jbrowse...

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by taspeotis on Monday August 26, 2013 @12:20AM (#6275172)
Character encodings are a pain in the ass. If you want some examples, Michael Kaplan posts [1] about these sorts of things in way too much detail.

[1] http://blogs.msdn.com/b/michkap/archive/tags/unicode+lame+li...

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by TheZenPsycho on Monday August 26, 2013 @06:11AM (#6276096)
this seems strangely similar to Joel Spolsky's article on unicode.

http://www.joelonsoftware.com/articles/Unicode.html

Though not exactly plagiarised, it shares some obvious parallels.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by derleth on Monday August 26, 2013 @12:46AM (#6275251)
> The original ASCII standard defined characters from 0 to 126.

0 to 127. 127 is a power of 2 minus 1, which should be a hint; in specific, it's two to the seventh minus one, since ASCII defines codepoints for all possible combinations of seven bits, which is 128 possible codepoints, so the enumeration ends at 127 if you count starting from zero, as computer programmers are wont to do.

> all possible 127 ASCII characters

128 characters, as mentioned above.

> the ASCII guys, who by the way, were American

ASCII stands for American Standard Code for Information Interchange. The ethnocentrism was unfortunate but it isn't like you weren't warned.

> The numbers are called “magic numbers” and they begin with U+.

He can call them "magic numbers" but everyone else calls them "codepoints".

> UTF-8 was an amazing concept: it single handedly and brilliantly handled backward ASCII compatibility making sure that Unicode is adopted by masses. Whoever came up with it must at least receive the Nobel Peace Prize.

I'm sure Ken Thompson and Rob Pike will be happy to hear someone thinks that way.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by lutusp on Monday August 26, 2013 @03:28AM (#6275660)
> 0 to 127. 127 is a power of 2 minus 1, which should be a hint; in specific, it's two to the seventh minus one ...

To expand on this a tiny bit, a power of two minus one is called a "Mersenne number". If the number is prime, it's called ... wait for it ... a Mersenne prime.

When expressed in binary, Mersenne numbers are an uninterrupted series of one digits: 11111... of varying lengths.

http://mathworld.wolfram.com/MersenneNumber.html

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by spacehunt on Monday August 26, 2013 @02:22AM (#6275482)
>> UTF-8 was an amazing concept: it single handedly and brilliantly handled backward ASCII compatibility making sure that Unicode is adopted by masses. Whoever came up with it must at least receive the Nobel Peace Prize.

> I'm sure Ken Thompson and Rob Pike will be happy to hear someone thinks that way.

If that's the case then I think the designers of GB18030 deserve it more, because they achieved an encoding that is able to map all Unicode codepoints while being backwards compatible with GB2312, which is itself backwards compatible with ASCII.

But seriously UTF-8 is like sliced bread after having dealt with we-thought-64K-is-enough-so-lets-all-use-16-bits UCS-2/UTF-16.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by Dylan16807 on Monday August 26, 2013 @03:02AM (#6275590)
And even more so, if the current 20.1-bit format turns out to be insufficient, UTF-8 is happy to expand to at least 31 bits, where UTF-16 would be forced to undergo very awkward redesign, such as dedicating an entire plane to double-surrogates.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by millstone on Monday August 26, 2013 @03:39AM (#6275680)
The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. UTF-16 does not suffer from that particular problem.

Of course, there's a long ways to go from "a sequence of valid code units" to "a valid string," but it's still relevant.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by KayEss on Monday August 26, 2013 @04:50AM (#6275881)
>UTF-16 does not suffer from that particular problem

what about surrogate pairs? You can't have only one 16 bit word for a pair and have a valid UTF-16 sequence. This problem is real easy to do if you substring a UTF-16 sequence naively.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by millstone on Monday August 26, 2013 @12:38PM (#6278101)
Right, there's still plenty of ways you can screw up a string. My point was merely that UTF-8 has one failure mode (invalid code units) that UTF-16 does not suffer from, and that has implications in encoding converters and APIs.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by lucian1900 on Monday August 26, 2013 @07:42AM (#6276390)
Invalid bytes is a feature. It makes it much easier to figure out the bytes you have are not actually UTF8.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by spacehunt on Monday August 26, 2013 @04:47AM (#6275869)
UTF-8 is already much better than the legacy double-byte encodings (Big5, GB2312, Shift-JIS etc etc) because, at least you can tell from any byte whether it is supposed to be a part of a code sequence, without having to start all the way from the beginning of the string.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by zokier on Monday August 26, 2013 @10:03AM (#6277071)
> The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8.

In what sort of situation that is a problem?

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by millstone on Monday August 26, 2013 @01:04PM (#6278275)
It can allow for security exploits, if the different layers of your stack handle invalid code units differently.

There's a related issue with "non-shortest forms:" it's possible to encode the NUL character '\0' without actually using a null byte. This means that (for example) "round tripping" UTF-8 data can introduce a null byte, allowing for security exploits when mixed with C-style string processing.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by gsnedders on Monday August 26, 2013 @09:14AM (#6276791)
This is untrue, see D79 and D89 of Unicode 6.2: an ill-formed UTF-16 string is one containing lone surrogates.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by millstone on Monday August 26, 2013 @12:56PM (#6278219)
Yes, that's an ill-formed string. As I wrote, there's a long ways to go from a "sequence of valid code units" to "a valid string."

But what I wrote is nevertheless correct, and matters. See for example the special requirements on UTF-8 processing in D93:

    If the converter encounters an ill-formed UTF-8
    code unit sequence...it must not consume the
    successor bytes as part of the ill-formed
    subsequence whenever those successor bytes
    themselves constitute part of a well-formed UTF-8
    code unit subsequence....For a UTF-8 conversion
    process to consume valid successor bytes is not
    only non-conformant, but also leaves the converter
    open to security exploits."
"Ill-formed code unit sequence" is not possible with UTF-16. That's an advantage. Also note the last line: the existence of ill-formed code units has security implications that are not present with UTF-16.

To be fair, UTF-16 has its own unique issues too, like endianness.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by yuhong on Monday August 26, 2013 @12:51AM (#6275261)
>ASCII stands for American Standard Code for Information Interchange. The ethnocentrism was unfortunate but it isn't like you weren't warned.

Yea, I think it reflects how much technology development comes from America even back then as well as now.

Re:Plain Text Doesn’t Exist: Unicode and encodings demystified by namsral on Monday August 26, 2013 @07:35AM (#6276364)
Rob Pike's story on how he and Ken Thompson invented UTF-8 over dinner:

https://plus.google.com/101960720994009339267/posts/Rz1udTvt...