News for hackers. Stuff that matters.
Re:A mistake that killed Japan's software industry? (2023) by wffurr on Tuesday November 12, 2024 @07:44PM (#42121704)
Many Kanji and Chinese glyphs are distinct. There are also many variants of Chinese, among them simplified and traditional. They don't have distinct Unicode codepoints due to Han Unification which tried to cram them all into UTF-16.

You have to know the intended locale or the text to disambiguate and select the correct glyphs.

Re:A mistake that killed Japan's software industry? (2023) by Onavo on Tuesday November 12, 2024 @07:47PM (#42121727)
Well, the world is on UTF-8 now, they can extend the standard and put the Japanese characters these. There's plenty of space since it's variable length.
Re:A mistake that killed Japan's software industry? (2023) by wffurr on Tuesday November 12, 2024 @08:17PM (#42121927)
Sure but "they" can't undo the billions of already authored documents and the thousands of overlapping fonts.
Re:A mistake that killed Japan's software industry? (2023) by hnfong on Wednesday November 13, 2024 @11:23AM (#42127409)
It's actually what they did. I heard from people involved in the CJK-unification project that the Japanese representatives insisted this be done as a condition for their participation.

Given that there were thousands of such characters, some characters slipped through and the same code point was used by both Chinese and Japanese.

The Japanese have been bickering about the handful of those cases ever since.

Re:A mistake that killed Japan's software industry? (2023) by syncsynchalt on Thursday November 14, 2024 @01:53PM (#42139740)
Weirdly, there _are_ separate codepoints for simplified vs traditional chinese characters, cf. 丟 (traditional) vs 丢 (simplified) [1]. This contributes to the annoyance of Japanese users, it feels like the unicode consortium went out of their way to mangle only their language.

[1] One major reason is that pre-Unicode charsets exist which encode both traditional and simplified characters, and one of the primary goals of Unicode is to support round-trip mapping from any charset into Unicode and back without loss of information.