News for hackers. Stuff that matters.
Re:How ASCII lost and unicode won by ygra on Tuesday July 09, 2013 @08:52AM (#6012377)
Yes, but that's the distinction between character sets and encodings. A character set is just that, a set of characters. In this sense Unicode is a superset of ASCII, Latin 1, and every other pre-existing character set. Encodings map characters to byte( sequence)?s and they differ a lot in that respect.

Practically both character sets and encodings prove ample opportunities to mangle input, either by replacing unavailable characters by ? or �, or by mapping to the wrong characters because the wrong encoding was assumed (or by throwing errors because the input made no sense).

Re:How ASCII lost and unicode won by tome on Tuesday July 09, 2013 @08:59AM (#6012403)
I mean when ASCII assigns a number to a character, Unicode assigns the same number to the character. UTF16 and UTF32 may encode those numbers differently from how ASCII encodes them, but they are nonetheless the same numbers.

Unicode is not an extension of EDBIC or Latin1 because characters under them have different codepoints than they do under Unicode.

[Please correct me if I'm wrong. I'm not an expert but this is how it seems to me]

Re:How ASCII lost and unicode won by DougBTX on Tuesday July 09, 2013 @09:17AM (#6012481)
> [Please correct me if I'm wrong. I'm not an expert but this is how it seems to me]

You're basically right, but the subtle point is that these comments are in response to lotsofcows who said, "Anything that talks Unicode can talk ASCII (although not vice versa)."

The implication of "talk" is that the numbers are being encoded. qznc was trying to clarify that lotsofcows's statement is only true if the Unicode system is using an encoding which is backwards compatible with 7-bit ASCII, such as UTF-8.

Re:How ASCII lost and unicode won by tome on Tuesday July 09, 2013 @09:27AM (#6012520)
Thanks, I see :) My comment wasn't particularly relevant.
Re:How ASCII lost and unicode won by VLM on Tuesday July 09, 2013 @09:27AM (#6012521)
"Unicode assigns the same number to the character."

Extremely close but not quite right for some weird corner cases. There's either a cool feature or hideous bug in the unicode design, depends how you look at it, that lets you compose multiple unicode codes together into one glyph. So you can say, "Gimme the code for A with a bar over it" or "Gimme the code for A, now gimme the code for put a bar over the last glyph". Even worse you can stack compose characters, so you can write "A with circle on top and squiggly underneath" at least five different ways in binary that should be rendered visually identical.

There is the one true normalization technique that most people use to convert what they consider poor unicode grammar into a standard form. Some people don't use it of course. And anytime you give users freeform input who knows what kind of crud they'll feed you. So you can never really assume any unicode string is normalized unless you personally normalized it yourself. Even then theoretically two normalized strings, concatenated, might or might not be normalized anymore (although this is often not much of a problem). And this strikes substring manipulation too.

A fun source of buffer overflows is normalization can shrink OR EXPAND a unicode string, in theory. So if you use one of those languages without variable length strings, look out. Or if the language tries to take care of it for you, this leads to weird memory fragmentation. Maybe a DOS/DDOS opportunity?

Which leads to philosophical argument you must decide in your code, if two bitstreams don't match, but you get matching glyph renderings, from the program's point of view are they a match or not? Depends if what your program is trying to do I guess. Most languages have a library to handle this. Writing your own unicode handler functions is a "here be dragons" moment.

Composing characters are also a fun source of swear words if you're trying to count the number of characters in a file. Something that renders literally identically will have an identical number of characters but somewhat varying number of bytes.

Other fun philosophical arguments are if you read in a un-normalized string and output a normalized string have you changed the string? Well, both no, and yes. I'm sure there's crypto steagongraphy implications.

Google for "unicode normalization form nfc" and stuff like that.

The first page I found with a good faq was:

http://www.macchiato.com/unicode/nfc-faq

Re:How ASCII lost and unicode won by ygra on Tuesday July 09, 2013 @09:43AM (#6012589)
Someone below noted already that grapheme clusters are, at least for dealing with humans, a much better thing to talk about than bytes or abstract characters (code points). The statement "Unicode assigns the same number to the character." was indeed correct, though. It's just that character is such an overloaded word (meaning either byte, code point or grapheme); but for Unicode at least it is synonymous with code point.
Re:How ASCII lost and unicode won by qw on Tuesday July 09, 2013 @03:49PM (#6014924)
AFAIK all Latin1 characters codepoints with Unicode too.
Re:How ASCII lost and unicode won by cygx on Tuesday July 09, 2013 @09:01AM (#6012411)
The ASCII coded character set is a subset of Unicode, which is not true for arbitrary codesets, even if their character repertoire is a subset of the Unicode one.
Re:How ASCII lost and unicode won by sixbrx on Tuesday July 09, 2013 @09:05AM (#6012429)
But character sets are not just sets of characters, they are assignments of abstract glyphs to integers, regardless of the storage representation of the integer.
Re:How ASCII lost and unicode won by ygra on Tuesday July 09, 2013 @09:17AM (#6012484)
Eep, funny how I can make the same mistake twice in ~five years. You are right, of course.

Character sets therefore could be modelled as a partial function c : Character ↛ ℕ, while the encoding is a partial function e : ℕ ↛ seq Byte. Something like that.

ASCII and Latin 1 are therefore both subsets of Unicode, even though only ASCII is a subset of UTF-8 while no comparable Unicode encoding exists for Latin 1 or any other encoding.