News for hackers. Stuff that matters.
Re:Mimic – abusing Unicode to create tragedy by wodenokoto on Friday October 23, 2015 @07:56AM (#10438007)
So, one might wonder why these homo-graphs have different code points. After all the French A and the English A share the same code point.

It's really difficult to do the right thing here. If Greek question marks share code point with semi-colon, it obstructs search and replace for question marks.

Subtle differences in how Japanese and Chinese are written has led to differently written characters sharing the same code point. It's nice that you can easily look up most Japanese characters in a Chinese dictionary and see how they are used in China, but it has become frustratingly hard to get subtleties in their written form right. The Chinese version may have the line strike through another line, while the Japanese only has it touching.

I honestly don't know how to go about posting how to same code points have different written forms!

But it seems like it would be nice if code editors warned about text outside ascii. You usually only want that in strings and comments.

Re:Mimic – abusing Unicode to create tragedy by AnthonyMouse on Friday October 23, 2015 @11:32AM (#10439227)
> It's really difficult to do the right thing here.

There is a fairly good solution to this for things like code editors and URLs or search strings in browsers: If a string contains a non-ASCII code point that is a homograph for an ASCII character, swap the text and background colors.

Re:Mimic – abusing Unicode to create tragedy by wodenokoto on Friday October 23, 2015 @02:58PM (#10440600)
I meant the right approach to designating code points.
Re:Mimic – abusing Unicode to create tragedy by Allan_Smithee on Monday October 26, 2015 @06:09AM (#10450410)
In today's lesson, wodenokoto learned that real life is infinitely more complicated than San Francisco startups would have you believe.
Re:Mimic – abusing Unicode to create tragedy by bcoates on Friday October 23, 2015 @01:25PM (#10439985)
The reason for a lot of homographs with ASCII is that there are old code pages that have entire non-US alphabets and punction sets in the 128-255 range, and regular ASCII in the 0-127 range.

It's a design goal of Unicode to support exact round-trip transformation with any code page ever in use in the real world, so they can't unify two characters that appear at different code points in a greek codepage without breaking things, even if they're always graphically indistinguishable in a font.

Re:Mimic – abusing Unicode to create tragedy by lazyjones on Friday October 23, 2015 @08:09AM (#10438054)
> It's really difficult to do the right thing here. If Greek question marks share code point with semi-colon, it obstructs search and replace for question marks.

Context is the key here. Greek text doesn't use the semicolon for other purposes and searching/replacing such single characters in source code is a terrible idea anyway (think comments, string literals...). So what is the prohibitive failure scenario here?

Indistinguishable (for humans) characters with different code points were a stupid idea, it's fine to abuse it in order to point out that fact.

Re:Mimic – abusing Unicode to create tragedy by TazeTSchnitzel on Friday October 23, 2015 @12:00PM (#10439432)
They're not necessarily indistinguishable. Line-break rules, directionality and typographical conventions (height/width/alternate glyphs) may differ between apparent homographs. And using the same code point would make distinguishing between, say, Latin and Cyrillic searches difficult. How would Google tell if you mean CCCP or СССР?
Re:Mimic – abusing Unicode to create tragedy by wodenokoto on Friday October 23, 2015 @09:50AM (#10438493)
I don't know enough about Greek to deeply comment on it. some people do consider using the same code point for apostrophe and citations problematic (it's definitely annoying when doing word segmentation) we also have breaking and non-breaking spaces, as well as tabs. We luckily didn't follow typewriter conventions and collapsed several alphabetic characters!

A semicolon is not considered sentence final, whereas a question mark usually is. This makes it easier for software to auto capitalize. So at least that's possible a use case.

There's also the possibility that Greek requires the top dot to be square or circle, meaning it might in fact have subtle differences in print.

Re:Mimic – abusing Unicode to create tragedy by thaumasiotes on Friday October 23, 2015 @02:27PM (#10440384)
> some people do consider using the same code point for apostrophe and citations problematic (it's definitely annoying when doing word segmentation)

And those people are a menace. "Ball bearings" is a single word with a space in the middle. The only way you're going to get reliable word segmentation is with a natural language parser and a lexicon with an entry for "ball bearing". At that point, you're already recognizing "aren't" as a word; the punctuation isn't really relevant.

On the other hand, if you're not particularly upset about messing up space-including words, there's no real reason to be upset about apostrophe-including ones either.

Re:Mimic – abusing Unicode to create tragedy by lazyjones on Friday October 23, 2015 @10:46AM (#10438884)
> A semicolon is not considered sentence final, whereas a question mark usually is. This makes it easier for software to auto capitalize. So at least that's possible a use case.

Such software will need and have a language setting anyway (e.g. for hyphenation). It doesn't have to and cannot rely on code points alone, so the characters (or rather, different uses of the semicolon) needn't have different codepoints.

Re:Mimic – abusing Unicode to create tragedy by retbull on Friday October 23, 2015 @03:12PM (#10440685)
What if someone is quoting something in a different language for a translation or similar?
Re:Mimic – abusing Unicode to create tragedy by DarkUranium on Monday October 26, 2015 @04:27PM (#10454215)
To be fair, that would break hyphenation as well if both languages use a Latin alphabet (give or take a few carons and such).
Re:Mimic – abusing Unicode to create tragedy by omaranto on Friday October 23, 2015 @12:01PM (#10439443)
There is no "French A" and no "English A", I believe both languages call their alphabet the Roman (or Latin) alphabet. So those really are the same letter.
Re:Mimic – abusing Unicode to create tragedy by thaumasiotes on Friday October 23, 2015 @01:37PM (#10440053)
The Chinese could call their writing "the Roman alphabet" too (they don't), and label 闩 "A". So what?
Re:Mimic – abusing Unicode to create tragedy by kps on Friday October 23, 2015 @04:41PM (#10441180)
The Chinese could do that, but they'd be wrong. ‘French A’ and ‘English A’ have undiverged continuity with ‘Roman A’. Chinese writing is neither descended from Roman nor an alphabet.
Re:Mimic – abusing Unicode to create tragedy by thaumasiotes on Friday October 23, 2015 @07:50PM (#10441841)
> ‘French A’ and ‘English A’ have undiverged continuity with ‘Roman A’.

This is true, but it's equally true that they both have undiverged continuity with 'Greek Α', which has its own code point.

Re:Mimic – abusing Unicode to create tragedy by Gigablah on Friday October 23, 2015 @08:26AM (#10438130)
> The Chinese version may have the line strike through another line, while the Japanese only has it touching.

Case in point: 每 (chinese) vs 毎 (japanese).

Re:Mimic – abusing Unicode to create tragedy by apricot on Friday October 23, 2015 @09:50AM (#10438488)
The problem is compounded by the fact that in Chinese, there are "traditional" and "simplified" versions of many characters (with the Japanese character usually, but not always, being the same as the traditional Chinese one). And let's not even get into the different writing styles. The distinction between different characters and different ways of writing the same character is not always clear.
Re:Mimic – abusing Unicode to create tragedy by rootbear on Friday October 23, 2015 @08:59AM (#10438264)
I had to zoom in to my screen just to see the difference between those.

I know that I am perhaps not well informed on this issue, but I also think that having multiple code points for what most people would perceive as the "same" character is not a good idea.

Re:Mimic – abusing Unicode to create tragedy by greggman on Friday October 23, 2015 @10:59AM (#10438974)
They are not perceived as the same by most people in those cultures. There are tons of characters that to someone who's eyes are not used to spotting the difference would not notice they are actually different. A single dot, or touch vs pass through or curved at the tip vs not make all the difference

入 vs 人

玉 vs 王

口 vs ロ <- my font might make those indisguishable out of context

千 vs 干 vs 于

of course in English we have 0 vs O and l vs 1

Re:Mimic – abusing Unicode to create tragedy by fryguy on Friday October 23, 2015 @02:23PM (#10440364)
In English, even within the same letter there are variations on how it can look. For instance, the letter a can look different based on serif/sans-serif, double-story/single-story, or even cursive and italic. There isn't that much variation in fonts for Asian characters, is there?
Re:Mimic – abusing Unicode to create tragedy by Nadya on Friday October 23, 2015 @02:27PM (#10440387)
There are also plenty of variations in calligraphy that make some store signs nearly impossible to read (for me). There are differences but many of them come from computers not being able to properly render pictographs.

Compare 人 with: http://i.imgur.com/ZpFMQoU.png for one of the most obvious differences.

For Japanese there is: 篆書、隷書、楷書、行書、草書.

https://en.wikipedia.org/wiki/Seal_script

https://en.wikipedia.org/wiki/Clerical_script

https://en.wikipedia.org/wiki/Regular_script

https://en.wikipedia.org/wiki/Semi-cursive_script

https://en.wikipedia.org/wiki/Cursive_script_(East_Asia)

Can see more here:

https://en.wikipedia.org/wiki/Japanese_calligraphy

Re:Mimic – abusing Unicode to create tragedy by Grue3 on Friday October 23, 2015 @09:23AM (#10438370)
It's a matter of the font size. At the size they're usually displayed, the characters look quite different [1] [2]. The real problem is that there's no way to mark a character as Chinese or Japanese on HN. So if it actually was the same character it would've been displayed identically. A common character that looks very different in Japanese and Chinese is 直 [3] and there's only one Unicode codepoint for it.

[1] https://en.wiktionary.org/wiki/%E6%AF%8F [2] https://en.wiktionary.org/wiki/%E6%AF%8E

[3] https://en.wiktionary.org/wiki/%E7%9B%B4

Re:Mimic – abusing Unicode to create tragedy by theoh on Friday October 23, 2015 @09:44AM (#10438460)
The case of the Turkish "i" is a good example of the difficulties faced by Unicode in this area. See multiple previous discussions on HN.
Re:Mimic – abusing Unicode to create tragedy by mamon on Friday October 23, 2015 @01:06PM (#10439862)
I was always curios about it: does reading Chinese requires better eyesight because glyphs are more complex?
Re:Mimic – abusing Unicode to create tragedy by thaumasiotes on Friday October 23, 2015 @01:40PM (#10440078)
No, it requires larger font sizes.
Re:Mimic – abusing Unicode to create tragedy by kneonx on Friday October 23, 2015 @11:15PM (#10442278)
Nope, it just requires you to know what the glyph means :o
Re:Mimic – abusing Unicode to create tragedy by mratzloff on Friday October 23, 2015 @09:28AM (#10438379)
It's not the same. It's written differently.
Re:Mimic – abusing Unicode to create tragedy by kps on Friday October 23, 2015 @10:24AM (#10438724)
So are single-story and double-story ‘a’ and ‘g’. But those don't have nationalist politics attached.
Re:Mimic – abusing Unicode to create tragedy by alextgordon on Friday October 23, 2015 @03:58PM (#10440958)
Think of them as words, not letters. It would supremely bother me if I were unable to write "colour" and always had to write "color".

I'm sure a Japanese person would be perplexed as to why that extra "u" matters since it's the "same damn word".

Re:Mimic – abusing Unicode to create tragedy by SixSigma on Friday October 23, 2015 @01:37PM (#10440054)
Come to England and go to the first floor of any building.

Want to jump off ?

Re:Mimic – abusing Unicode to create tragedy by dragonwriter on Friday October 23, 2015 @04:44PM (#10441201)
> But it seems like it would be nice if code editors warned about text outside ascii. You usually only want that in strings and comments.

That's somewhat language dependent, though its true for most uses of most popular languages.

Re:Mimic – abusing Unicode to create tragedy by ant6n on Friday October 23, 2015 @08:07AM (#10438043)
now that we gave up on ucs-2, we could re-encode those overloaded Japenese/Chinese characters as separate Japenese and Chinese characters on astral planes like the supplemental multilingual one.
Re:Mimic – abusing Unicode to create tragedy by rspeer on Friday October 23, 2015 @11:22AM (#10439144)
They're doing that. That's what a lot of plane 2 is.