News for hackers. Stuff that matters.
Re:Unitools – A suite of tools for working with Unicode in t... by dghf on Wednesday March 02, 2016 @11:35AM (#11210991)
Can you explain?

UTF-8 was deliberately designed to be "backwards compatible" (for want of a better phrase) with ASCII.

Absent a BOM mark, there is no way to tell the difference between a piece of ASCII text on the one hand and, on the other, a piece of UTF-8 text that uses no code point above U+007f. How do they differ semantically?

Re:Unitools – A suite of tools for working with Unicode in t... by khedoros on Wednesday March 02, 2016 @03:21PM (#11212698)
Loading it from disk, there's no way to tell the difference, because the representation is the same and a selection of bytes are a selection of bytes.

Say you read the text into a program. You can load the text into an AsciiString variable or a UnicodeString variable, and it's valid in either of those cases. They have the same binary representation...but they're semantically different. They each have properties that the other one doesn't (ASCII has random access, Unicode is linear access, either one could have an ASCII string appended to it, but only the Unicode one could have a Unicode string appended, etc).

The difference isn't the representation; the difference is the interpretation that we attach to the representation.

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @11:44AM (#11211072)
Yes. I already tried to explain with an analogy about faithfulness (which I removed because it got many downvotes).

When you deal with ASCII, that's also the semantic level on which you operate. Much simpler than Unicode (I don't think there are many people who really understand Unicode).

When you have a Unicode string that uses only characters that are also present in ASCII, that doesn't make it an ASCII string. It has more meaning.

Btw that has nothing to do with BOM. BOM is itself not part of ASCII, and not encoded as a single byte, and it doesn't affect single-byte sequences.

Re:Unitools – A suite of tools for working with Unicode in t... by emddudley on Wednesday March 02, 2016 @12:37PM (#11211504)
Are you talking about how, when someone is limited to ASCII, he makes substitutions he wouldn't make if he were using Unicode? Such as using :-) instead of U+263A?

Or are you maybe talking at a software level, where you can make some simplifications and assumptions about a string when it is guaranteed to be ASCII?

Because otherwise I can't see how there could be any semantic or meaningful difference between ASCII and Unicode when the text is the same. An ASCII 'a' has exactly the same meaning as a Unicode 'Latin Small Letter A'.

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @01:19PM (#11211816)
You can look at it from an information theoretic standpoint. The information you get from the string is also all the things that it does not contain.

Btw. I have no clue why I am constantly downvoted. Whoever did this, I don't think there is an incentive to downvote my two ancestor comments. It's demotivating.

Re:Unitools – A suite of tools for working with Unicode in t... by coldtea on Wednesday March 02, 2016 @02:20PM (#11212250)
Didn't downvote but I can see why they got downvoted: they make a claim, but when asked for a specific argument, they go roundabout.

In what SPECIFIC way is an only-ascii string not a UTF-8 string for ALL intends and purposes?

You throw some words like "semantics" but no actual arguments.

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @03:47PM (#11212885)
I think I answered that in the clearest way I (personally) could, right in the comment to which you replied.
Re:Unitools – A suite of tools for working with Unicode in t... by dghf on Wednesday March 02, 2016 @12:22PM (#11211358)
But wherein does that meaning lie? If the two strings represent the same text, you can't tell the difference between them, because there is none.

(I guess technically you can't tell the difference between an ASCII string and a string of random 8-bit numbers in the range 0 <= x <= 127, but if when represented as ASCII characters those numbers are meaningful -- comprehensible English, for example -- the odds are good it's the former not the latter.)

Re:Unitools – A suite of tools for working with Unicode in t... by technofire on Wednesday March 02, 2016 @12:35PM (#11211488)
I think what he's getting at is that whatever program is reading the text must be prepared to read UTF-8 and not just ASCII. Sure, the underlying data is the same, but the data type is not the same; analogously, a byte with the decimal value of 65 might be the number 65 if cast as an integer, but might be the character 'A' if cast as a char.