Actually yes, the CJK unification is a problem for many people, including me when I want to read Japanese on a phone bought in Europe.

Example 1. Typically, any time you want to mix the 2 languages you're getting in trouble.

Let's say you write a textbook for Chinese people to learn Japanese as a second language. Or a research article in Japanese citing old Chinese literature.

In your text, you'll have to mark specifically which part are in Japanese and which part are in Chinese and use different font for them. If you don't, the characters look wrong.

Example 2. Most phone software don't switch font for languages, so they pick one. Let's say you buy a Japanese phone, it's a Japanese font everywhere. For some reason on a Samsung or Motorola phone bought in Europe, it's a Chinese font everywhere and it looks ugly for Japanese. Wasn't Unicode supposed to be universal, and allow you to read in any language with a device bought anywhere?

On the other hand, I don't think the "I can't write my name" is really a problem for Japanese people, or at least it's not a Unicode problem. Some people have rare characters in their names. They accept and understand not being able to type their name on a computer and having to settle for a close characters. Unicode actually provided more characters than previous Japanese-only encodings like JIS so there are people who could write their name thanks to Unicode.

> In your text, you'll have to mark specifically which part are in Japanese and which part are in Chinese and use different font for them. If you don't, the characters look wrong.

Most non-trivial text-rendering systems need to know the language of the text in order to properly render it. This is not only an issue for Japanese/Chinese but also e.g. English/German with different hyphenation rules[0], different rules for ligatures[1] and different rules/preferences for spacing after punctuation[2]. None of these can sensibly be solved by Unicode.

[0] ‘backen’ (to bake) becomes bak-ken when hyphenated at the end of a line, ‘airlock’ can’t be hyphenated at all between c and k.

[1] ‘Schiff’ (ship) should be rendered with a ligature ff, ‘Dampfflut’ (~ steam flow?) shouldn’t. ‘afferent’ probably should have it.

[2] Single space after ‘.’ and ‘,’ in German, whereas French tends to prefer more space after ‘.’.

Neither of your examples are solveable by the Unicode consortium, who are not the god-emperors of fonts (nor would you want them to be).

I respectfully disagree. If Japanese ideograms and Chinese ideograms actually used different code points (i.e. no "Han unification"), then the problem wouldn't exist - the phone could trivially use a Japanese font for Japanese text, and a Chinese font for Chinese text.

No. Using different code points for the same character used in different languages creates big problems. It would be like having different code points for 'A' depending on whether it was used in English, Spanish, German, etc. If you somehow ended up writing "color" with both 'o' characters from the Spanish ABCs and the others from the English ABCs, you'd have a real mess when it came to sorting, searching, name matching (what language is "Hans"?) etc. It is far more convenient to allow the character sequence "color" or "Hans" to be language independent, even if the font choices, pronunciation, sort order, etc., are language dependent.

Chinese, Japanese, and Korean writers face similar issues. The characters they use to write the name of China or Japan, the ten digits, the characters for year, month, and day in dates, and so many thousands of others in Chinese characters are what they all consider to be the same characters. That is not all characters, but it includes so many that insisting on different code points by language would make a real mess. Hong Kong has many characters that are unique to HK Cantonese. So, should Cantonese have a full set of all Chinese characters that are the "Cantonese characters"? How about Shanghainese, then? Or Hakka? Teochew (Chaozhou) or a dozen Chinese languages? Full, independent sets of all Chinese characters for each? Suppose you accidentally used an input method in HK and wrote the name of some Beijing gov't ministry using characters that looked identical to their Mandarin counterparts but were entirely different codepoints? Now what? You can't find your search term? You mess up the database and have two identical-looking keys?

No, Han unification is not conceptually different from unifying ABCs used by English and Spanish speakers, Cyrillic used by Russians or Serbs, etc., except that there are many more characters, so the boundary between what should be unified and what shouldn't contains more items in the gray zone to cause debate. Having no Han unification at all wouldn't solve all problems, it would create all sorts of absurdity.

Sure, having no unification at all would be bad, but the issue is with the gray zone. Some characters are written identically in each CJK language, but among those that aren't the amount of difference varies widely. The trouble is that Unicode leaves separate codepoints for each version that somebody, somewhere decided were "different enough" (even when they are the same character historically and linguistically) but merges many characters with (consistent, well-defined) differences because somebody felt they were close enough for horseshoes. People often think that characters were only merged if they were linguistically the same, but that's not the case.

Also, comparisons like "different ABCs for English and Spanish" are spurious and unhelpful. If you could tell an English "b" from a Spanish one by looking at it, the comparison would be sound.

Actually, Cyrillic is an interesting case. The Unicode standard does define completely-separate codepoints for the Cyrillic letters, even for the ones that look "just like" letters of the Latin Alphabet. Greek letters that look exactly like Latin letters get the same treatment.

It's difficult to come up with a logical explanation for why European languages that use their own alphabet get their own codepoints, but ideographic languages need to be "unified", even though the actual letters as used in those languages look different.

The "Han unification" was fundamentally a bad idea, and persists for historical reasons. Back when (some) people thought a fixed-width 16-bit character representation would be "enough", it made sense to try to reduce the number of "redundant" code points. Now that Unicode has expanded to a much-larger code space, I would think they'd choose differently.

Unfortunately, that kind of sweeping change is unlikely any time soon.

When you phrase it this way, "using different code for the same characters" it sounds obvious, but the problem precisely is whether they are the same characters or not. Are they like sans serif (used in English) or Gothic (traditionally used in German), or are they like the Roman alphabet and the Cyrillic alphabet, different but from the same root?

Gather 5 linguists, you won't get them to agree on that. Unicode says they're the same, but not everyone agrees with them, and the practical problems are real.

But.. you already posted a solution to your problem - use chinese font for chinese and japanese font for japanese. There is no problem. I mean, you would also presumably use a western font for latin and another separate font for cyrillic, since japanese fonts universally have absolutely dreadful kerning on latin (and often omit cyrillic entirely).

Having to know metadata about text is precisely what made the pre-unicode days so bad. If you're writing a word document, sure, choose your fonts, but if you're rendering a web page that doesn't happen to declare its language, things aren't so simple. (And if you're writing software meant to correctly handle user input in multiple languages, good luck...)

I run into this problem quite often. Has anyone proposed creating an alias for each character that indicated that it was chinese, japanese, or korean? In this way you could mix words within documents and when it is rendered each language could be represented by different fonts? The same could be used for variations in chinese -- so that if you used a Hong Kong Cantonese or Taiwanese namespace or alias it would use traditional characters, and if using a Mandarin, or Singapore Chinese namespace it would use simplified characters.

I agree, but it seems tricky like it would be tricky to strike exactly the right balance between unifying too much and too little.

The arguments for Han unification could have just as well been applied to unifying the Nordic languages – the Swedish Ä is really exactly the same letter as Danish/Norwegian Æ (except that Swedish words never ever use the latter and presumably Danish words never use the former), so it could be argued that it should have the same codepoint, forcing us to use country-specific fonts and making it impossible to use both languages in a simple text file.

So, for example, running "git log" on a project with an international contributor base would render either Swedish och Danish names incorrectly, depending on which font the user has selected. That would look extremely strange to me, so I imagine Chinese/Japanese/Korean users feel similarly.

(Luckily for us Nordic users, our letters were already separate characters in ISO-8859-1, which Unicode is a superset of.)

Ä and Æ are more different from eachother than the characters in Chinese and Japanese which have been merged. They are used for the same thing, they share etymology, but they are not the same letter.

By the way, Unicode is about scripts, not languages. If we started distinguishing by language, we might need to start remembering that china doesn't have a single language. Duplicating all those characters again to cover Mandarin and Cantonese and Wu and Hakka and Hokkien?

Should we have a different letter for Æ in Bokmål and Nynorsk? How about Riskmål?

The equivalent of Æ and Ä in Han scripts have not been unified. The equivalent of a in Futura and a in Verdana have. There's tons of chinese characters, so a few of them are bound to be borderline and maybe contentious. But overall, Han unification was unavoidable for a project like Unicode.

> Ä and Æ are more different from eachother than the characters in Chinese and Japanese which have been merged.

> They are used for the same thing, they share etymology, but they are not the same letter.

That second quote applies equally to Ä and Æ.

It applies to Ä and Æ... which is what the parent said. It doesn't apply to 気 and 气 and 氣. Those are all the same thing.

気　气 and 氣 are actually not merged by han unification, and could be described as similar to similar to Ä and Æ: shared etymology, same meaning, some languages decided to use a simpler character because the old one was too complicated to write.

The variations on characters which have been merged are usually even closer than that. More like the single or double storey "a", or the single or double loop "g".

Well, maybe if the double-story "a" were used exclusively in one country but considered incorrect in another.

I think what you're missing is that Han unification never particularly concerned itself with whether characters are "the same" or not. Somebody basically just decided that some differences were important and others weren't. I mean, the difference between 語 and 语 is analogous to printing and cursive, but for whatever reason it made the cut. Meanwhile the SC and TC versions of 骨 are approximately mirror images, but to show you I'd need two different fonts.

Anyway the merged differences are not just analogous to different fonts.

> I agree, but it seems tricky like it would be tricky to strike exactly the right balance between unifying too much and too little.

That may be true, but not unifying at all must certainly fall under unifying too little.

Not to say there aren’t problems with CJK unification. It’s difficult or impossible to represent old family names or newly coined characters without some means of composing characters from radicals—even if you do want “precomposed” characters a majority of the time, as is the case with, say, é (00E9) versus é (0065 0301).

As far as these characters being “the same”, I think it’s better to say they’re analogues. A in English is analogous to A in Polish—both are Latin alphabets, but neither is the Latin alphabet, and these languages use “the same” letter quite differently. And, say, Å is a letter in Swedish, but in English it’s A with a separate accent mark.

These are also just as well analogous to Α in Greek or А in Cyrillic, so hell, I would actually support the author’s “Greco Unification” strawman if it could be done in a principled way.

> I would actually support the author’s “Greco Unification” strawman if it could be done in a principled way

A 5 year old tech note by a Unicode Consortium member at http://www.unicode.org/notes/tn26 gives 7 reasons "why the Latin, Greek, and Cyrillic scripts have been separately encoded, rather than being encoded as a single script", and 12 reasons why the Han script was unified.

Edit: These 7 reasons make a good case why such "Greco Unification" wouldn't work.

I'm not convinced that this isn't just post-hoc justification.

All seven reasons boil down to "It's always been done this way".

I don't see how "a" in the Latin alphabet as used in English is any different to the "a" in the Latin alphabet used in Polish - in particular they are rendered in the same way.

Yet the article claims that even though Chinese and Japanese versions of characters now appear wildly different they still represent the same script so bad luck. Should the Unicode consortium be concerned with the glyphs or the semantics? Seems they are selectively doing both.

Very good points. That’s why I said “if it could be done in a principled way”—I don’t think it can.

I'm having trouble finding the reference now, but years ago I did read a story about how an early version of Unicode's reference document depicted CJK characters in a font with Chinese-styled characters, causing slowed adoption in Japan who had a gut reaction that "Unicode is too Chinese". Later editions printed several variations of characters, and the objections mostly evaporated.

(Apologies if I've gotten the countries reversed in this story, it's been years and google is failing me)

> [...] CJK unification[...] has never been a point of contention in the communities concerned with it.

I am not very familar with the CJK unification project so take my points with a grain of salt.

> more than not opposing CJK unification, I benefit from it greatly.

I think think that is a different point of view. Isn't it? You are seeing your benefit whereas the author is seeing his. Here's an alternative solution: What if the search engine understood what you were searching for and returned results in all the languages? Unification can result in a lot of information loss the same way a photo can be compressed but it comes at the cost of loss in quality.

> so there is(to my eyes at least) no value in fragmenting instances of the same character.

But no-one is fragmenting instances of the same character. They _are_ different characters from different languages. To take an example from the article, I am not sure how I feel about combining B and β. You are either ignoring the whole of English speaker population or the greek speaking one. Given that you have complete flexibility to assign a code for both of them, why not do it(responsibly)?

No experience in Asian languages, but J is pronounced differently in English and German. Should they have unique characters?

The example characters expressed: 日、中、力 however are written the same way in both Chinese and Japanese from my understanding. (Albeit, I studied Japanese).

There are admittedly variations which should be done separately, however unification of visually identical glyphs is a "good thing" imho

I'm fluent in Japanese and speak some Mandarin Chinese as well. These 3 characters are identical, not similar.

For a different example, 国 and 國 used to be the same character, but China and Japan (left) have both diverged the traditional form still used in Taiwan (right). Unicode treats them as separate.

今 Looks slightly different in traditional Chinese vs other languages. In traditional Chinese, the little straight line between the two angled lines is sloped, while it is horizontal in simplified Chinese, Japanese or Korean. Any reader of any of these languages would have no issue if the variant they are used to was replaced by the other one. They might think you have a sloppy handwriting or an ugly font if they even notice, but that's about it. Unicode treats them as the same.

This seems again to be a perfect place for rendering rather than encoding. The english letter 'a' can be rendered as a ring with a tail (the way I handwrite), or a ring with a cap and a tail (the way the font usually renders). Both are the same letter, if rendered differently based on my (contextually sensitive) font.

I think not. I do want to be able to say both People's Republic of China 中华人民共和国 and Republic of China 中華民國 in the same text, and if I had to choose rendering either 国+华 or 國+華 then it wouldn't work.

Curious, as I'm not sure when that would actually happen in real life (in Chinese). Generally in mainland China , the ROC would always be rendered with 国, even officially [1]. And in Taiwan the PRC would be rendered with 國 [2].

It gets a bit weirder in Japanese where the word is distinctly not the same - one is a traditional version (proper noun) of the other and you could imagine a text using both (William vs Wilhelm vs Will).

[1] http://baike.baidu.com/view/2200.htm [2] https://www.google.com.tw/?gws_rd=ssl#q=%E4%B8%AD%E5%8D%8E%E...

That is true, mainland China writes "中华民国".

But I still do want to be able to write texts that are like this discussion: mainly in English, but contain fragments in Chinese, and so that I can use both the traditional and simplified characters.

And it also makes total sense to me that 日本 is Japan, both in Japanese and Chinese, using the exact same Unicode characters.

If it's up to font rendering, you would specify the language tag for each of those individually which would render the appropriate font (as you are writing that country's name in its language).

Though that's far more difficult for the laman, and markdown certainly doesn't seem to have any language markers.

Except you can still recognize the 'a' as 'a' no matter which way it is rendered.

Not so with Chinese characters. For instance, the character for "fly" in simplified (飞) and traditional (飛) look very different. Someone who only learned simplified may not recognize the traditional character as being the same.

Which is exactly why 飞 and 飛 are encoded separately. I don't see any problem with that.

Yes, but other characters that also look different are merged. Here's an example: http://www.tofugu.com/2012/04/04/the-sorry-state-of-japanese...

That's the character for "cold". If you showed me (a Chinese speaker) the Japanese or Korean variant, I would have no idea what it meant.

The fact that some characters are debatable does not change the fact that han unification is a good idea. In a few cases you can disagree, but not unifying at all would be madness.

As for this particular character (cold), both variants are familiar to Japanese readers, with the one described as Japanese in your link being the one you'd typically see in print, while the other one is common in handwriting, and nobody in Japan would treat these two as different. From a Japanese point of view, this is definitely the kind of thing you change by switching fonts.

This pdf is the official list of basic chinese characters, published by the Japanese governement. Look on page 9, it shows both variants of this characters in hand writing.

http://www.bunka.go.jp/kokugo_nihongo/pdf/jouyoukanjihyou_h2...

The fact that one of the variants in not familiar to Chinese readers complicates the issue, but there are at least reasonable reasons to argue that this is one, not two, characters.

I think it is possible to argue that han unification was not done very well and that the UC made too many classification mistakes (although I personally think it is generally not that bad), but I don't think arguing that unification is a bad thing entirely has legs.

As far as I can see, Unicode has mostly settled down into a sort of "good enough" state: characters that have sufficiently different renderings have gotten separate "variant" codepoints for each rendering, while characters that are very similar (even if not completely identical as commonly written) are still only present as unified codepoints.

I've no idea if these variant codepoints are actually supposed to show up in user files, or are intended mainly for the use of font rendering systems, etc... the whole thing seems a bit of a mess, even if the information is technically present.

Judging from unicode.com, "冷" does seem have separate codepoints: 冷 (chinese/unified), and 冷 (japanese z-variant). However my browser renders both as similar characters. Similarly, on my phone, the same character gets input whether using a Chinese or a Japanese input method, and both get rendered using the Japanese rendering (it's a Japanese phone) which makes Chinese text look a little funny.

An interesting example is "晩" / "晚", which has one more stroke in the Japanese variant, but it's situated in a location which makes both variants look pretty much identical (and in small bitmapped fonts, they are identical). Nonetheless, Unicode includes codepoints for both...

Encoding those as the same code point makes sense. On the other hand, just the codepoint is obviously not enough on its own for rendering the glyph. The Unicode consortium seem not to care about the actual rendering part of the whole stack and are happy just defining the low-level bits. But then why do we have skin colour coding for emoji and no language coding for CJK glyphs? The entire thing is a mess, but heaping another pile of standards on top of it will make it even more of a mess, I'm afraid.

I would like to take a moment to have people recognize that there are differences between the glyphs used for these characters, and some people find this to be a point of contention.

For example, in the case of the text fragment “福祉”, there are three glyph representations for each character. Referenced below is an image demonstrating these three; first is Chinese, second is Korean, third is Japanese. [0] (I left out the Taiwanese etc. because they don't differ in this case from the CN)

The UC have clear rules for this, as explained in their section "Characters, not glyphs", which starts on page 8 of the chapter 2 pdf of the Unicode 7.0 standard. They make the case that codepoints are not for representing stylistic or glyphic distinctions, as seen here, but rather for semantic separations; that is, people still agree that these are the same characters, even if they typeset and write them differently in Chinese, Korean, and Japanese text.

They make cases specifically against a greco-latin unification, and for the CJK unification in Technical Note #26[1]

[0] http://marumie.magnifi.ca:8080/ipfs/QmXDnJUPVtzxVtAQgFkNHwvk... [1] http://www.unicode.org/notes/tn26/

I think it is a problem when 2 characters that are written differently share the same code point.

Having 大 be the same code point for Chinese, Japanese and Korean is to me as obvious as having the same A for English and Italian. We agree on this point.

But some of the unified characters listed on Wikipedia ought to have different code points because they are no longer written the same across languages.

https://en.wikipedia.org/wiki/Han_unification#Examples_of_la...

This is like having a and α share the same code point.

You're oversimplifying this.

Take characters like 今 or 骨. Whether the C/J/K variants are the "same characters" or not is semantic, but philosophy aside they are written differently in each language. That is to say, if you are localizing an app into a CJK language, your app would not be working correctly if it displayed Japanese text with a Chinese 今, or vice versa. If you showed it to a user you'd get a bug report.

But such characters have only one codepoint, so unless you have language metadata for your text, you can't render it correctly (which is half of was so bad about the pre-unicode days!).

(Note: Unicode later added "variation selectors", which as far as I know solve the problems mentioned above. I don't know why the aren't in widespread use, or perhaps they are and I'm misunderstanding something.)

>There should not be three codepoints for 中, 日, 力, and most other characters

There wouldn't be. Where the characters do not diverge, you don't need separate codepoints, surely?