News for hackers. Stuff that matters.
Re:How ASCII lost and unicode won by tome on Tuesday July 09, 2013 @08:59AM (#6012403)
I mean when ASCII assigns a number to a character, Unicode assigns the same number to the character. UTF16 and UTF32 may encode those numbers differently from how ASCII encodes them, but they are nonetheless the same numbers.

Unicode is not an extension of EDBIC or Latin1 because characters under them have different codepoints than they do under Unicode.

[Please correct me if I'm wrong. I'm not an expert but this is how it seems to me]

Re:How ASCII lost and unicode won by DougBTX on Tuesday July 09, 2013 @09:17AM (#6012481)
> [Please correct me if I'm wrong. I'm not an expert but this is how it seems to me]

You're basically right, but the subtle point is that these comments are in response to lotsofcows who said, "Anything that talks Unicode can talk ASCII (although not vice versa)."

The implication of "talk" is that the numbers are being encoded. qznc was trying to clarify that lotsofcows's statement is only true if the Unicode system is using an encoding which is backwards compatible with 7-bit ASCII, such as UTF-8.

Re:How ASCII lost and unicode won by tome on Tuesday July 09, 2013 @09:27AM (#6012520)
Thanks, I see :) My comment wasn't particularly relevant.
Re:How ASCII lost and unicode won by VLM on Tuesday July 09, 2013 @09:27AM (#6012521)
"Unicode assigns the same number to the character."

Extremely close but not quite right for some weird corner cases. There's either a cool feature or hideous bug in the unicode design, depends how you look at it, that lets you compose multiple unicode codes together into one glyph. So you can say, "Gimme the code for A with a bar over it" or "Gimme the code for A, now gimme the code for put a bar over the last glyph". Even worse you can stack compose characters, so you can write "A with circle on top and squiggly underneath" at least five different ways in binary that should be rendered visually identical.

There is the one true normalization technique that most people use to convert what they consider poor unicode grammar into a standard form. Some people don't use it of course. And anytime you give users freeform input who knows what kind of crud they'll feed you. So you can never really assume any unicode string is normalized unless you personally normalized it yourself. Even then theoretically two normalized strings, concatenated, might or might not be normalized anymore (although this is often not much of a problem). And this strikes substring manipulation too.

A fun source of buffer overflows is normalization can shrink OR EXPAND a unicode string, in theory. So if you use one of those languages without variable length strings, look out. Or if the language tries to take care of it for you, this leads to weird memory fragmentation. Maybe a DOS/DDOS opportunity?

Which leads to philosophical argument you must decide in your code, if two bitstreams don't match, but you get matching glyph renderings, from the program's point of view are they a match or not? Depends if what your program is trying to do I guess. Most languages have a library to handle this. Writing your own unicode handler functions is a "here be dragons" moment.

Composing characters are also a fun source of swear words if you're trying to count the number of characters in a file. Something that renders literally identically will have an identical number of characters but somewhat varying number of bytes.

Other fun philosophical arguments are if you read in a un-normalized string and output a normalized string have you changed the string? Well, both no, and yes. I'm sure there's crypto steagongraphy implications.

Google for "unicode normalization form nfc" and stuff like that.

The first page I found with a good faq was:

http://www.macchiato.com/unicode/nfc-faq

Re:How ASCII lost and unicode won by ygra on Tuesday July 09, 2013 @09:43AM (#6012589)
Someone below noted already that grapheme clusters are, at least for dealing with humans, a much better thing to talk about than bytes or abstract characters (code points). The statement "Unicode assigns the same number to the character." was indeed correct, though. It's just that character is such an overloaded word (meaning either byte, code point or grapheme); but for Unicode at least it is synonymous with code point.
Re:How ASCII lost and unicode won by qw on Tuesday July 09, 2013 @03:49PM (#6014924)
AFAIK all Latin1 characters codepoints with Unicode too.