News for hackers. Stuff that matters.
Re:The ü/ü Conundrum by lmm on Sunday March 24, 2024 @07:06PM (#39811348)
> Can we please talk about Unicode without the myth of Han Unification being bad somehow?

It's not a myth, as anyone living in Japan knows, and the "just use Unicode, all you need is Unicode" dogma is really harmful; a lot of "international" software has become significantly worse for Japanese users since it took hold.

> The problem here is exactly the lack of unification in Roman alphabets!

Problems caused by failing to unify characters that look the same do not mean it was a good idea to unify characters that look different!

Re:The ü/ü Conundrum by eviks on Sunday March 24, 2024 @11:25PM (#39812611)
Would it still be harmful if language tag were used?
Re:The ü/ü Conundrum by lmm on Sunday March 24, 2024 @11:37PM (#39812664)
If the tag mechanism was used consistently and handled by all software, no. But in practice the only way that would happen is if the tag mechanism was required for many languages. Unicode is, in practice, a system that works the same way for ~every human language except Japanese, which makes it much worse than the previous "byte stream + encoding" system where any program written to support anything more than just US English would naturally work correctly for every other language, including Japanese.
Re:The ü/ü Conundrum by renhanxue on Monday March 25, 2024 @08:51AM (#39815680)
> Unicode is, in practice, a system that works the same way for ~every human language except Japanese

This is simply not true. As I've pointed out in a sibling comment, Unicode has a lot of surprising and frustrating behaviors with many European languages as well if you use it without locale data. The characters will look right, but e.g. searching, sorting and case-insensitive comparisons will not work as expected if the application is not locale aware.

Re:The ü/ü Conundrum by lmm on Monday March 25, 2024 @06:40PM (#39822050)
> The characters will look right, but e.g. searching, sorting and case-insensitive comparisons will not work as expected

This is quite a different situation from Japan. A lot of applications don't do searching, sorting, or case-insensitive comparisons, but virtually every application displays text.

Re:The ü/ü Conundrum by jhanschoo on Monday March 25, 2024 @02:04AM (#39813171)
> "just use Unicode, all you need is Unicode" dogma is really harmful; a lot of "international" software has become significantly worse for Japanese users since it took hold.

The alternative would be that the software used Shift_JIS with a Japanese font. If the software used a Japanese font for Japanese it wouldn't need metadata anyway.

There really isn't a problem with Han unification as long as you always switch to a font appropriate for your language; you don't need to configure metadata. If you don't you are always going to run into missing codepoint problems.

In cases where the system or user configures the font, properly using Unicode is still easier than configuring alternate encodings for multiple languages.

Re:The ü/ü Conundrum by lmm on Monday March 25, 2024 @02:18AM (#39813229)
> The alternative would be that the software used Shift_JIS with a Japanese font.

As far as I know all Shift_JIS fonts are Japanese; you would have to be wilfully perverse to make one that wasn't.

> If the software used a Japanese font for Japanese it wouldn't need metadata anyway.

If it just uses the system default font for that encoding, as almost all software does, then it will also behave correctly.

> There really isn't a problem with Han unification as long as you always switch to a font appropriate for your language

Right. But approximately no software does that, because if you don't do it then your software will work fine everywhere other than Japan, and even in Japan it will kind-of-sort-of work to the point that a non-native probably won't notice a problem.

> In cases where the system or user configures the font, properly using Unicode is still easier than configuring alternate encodings for multiple languages.

I'm not convinced it is. Configuring your software to use the right font on a Unicode system is, as far as I can see, at least as hard as configuring your software to use the right encoding on a non-Unicode system. It just fails less obviously when you don't, particularly outside Japan.

Re:The ü/ü Conundrum by GoblinSlayer on Monday March 25, 2024 @09:29AM (#39816041)
Japanese programs aren't globalized and already rely on the system being fine tuned for Japanese, so default font is already correct.
Re:The ü/ü Conundrum by lmm on Monday March 25, 2024 @06:37PM (#39822031)
> Japanese programs aren't globalized and already rely on the system being fine tuned for Japanese

Right, because unicode-based systems don't work well in Japan. E.g. a unicode-based application framework that ships its own font and expects to use it will display ok everywhere that's not Japan. So Japan is increasingly cut off from the paradigms that the rest of the world is using.

Re:The ü/ü Conundrum by GoblinSlayer on Tuesday March 26, 2024 @04:26AM (#39825348)
Custom fonts are often a mistake for any language, especially google fonts often look wrong. Due to this browsers often have an option to force usage of system fonts and set minimum size to improve readability.
Re:The ü/ü Conundrum by lmm on Tuesday March 26, 2024 @05:55PM (#39833311)
> Custom fonts are often a mistake for any language, especially google fonts often look wrong.

Be that as it may, the overwhelming majority of unicode fonts are dramatically wrong for Japanese and not dramatically wrong for other languages.

> Due to this browsers often have an option to force usage of system fonts and set minimum size to improve readability.

Such options are shrinking IME. E.g. Electron is built on browser internals, but does it offer that option?

Re:The ü/ü Conundrum by jhanschoo on Wednesday March 27, 2024 @12:04AM (#39835608)
> Right. But approximately no software does that, because if you don't do it then your software will work fine everywhere other than Japan, and even in Japan it will kind-of-sort-of work to the point that a non-native probably won't notice a problem.

Most games that I know of that target CJK + English (and are either CJK-developed, or have a local publisher based in East Asia) do indeed switch fonts depending on language (and on TC vs. SC).

> I'm not convinced it is. Configuring your software to use the right font on a Unicode system is, as far as I can see, at least as hard as configuring your software to use the right encoding on a non-Unicode system. It just fails less obviously when you don't, particularly outside Japan.

I'm considering 3 scenarios:

1. You are configuring for the Japanese-speaking market. In which case, fix a font, or fonts.

2. You are localizing into multiple languages and care about localization quality. In which case, yes, you need to know that localization in Unicode is more than just replacing content strings, but this is comparable to dealing with multiple encodings.

3. You are localizing into multiple languages and do not care about localization quality, or Japanese is not a localization target. In which case Japanese (user input / replaced strings) in your app / website will appear childish and shoddy, but it is still a better experience than mojibake.

In any case, it seems to me that it is not a worse experience than pre-Unicode. It's just that people who have no experience in localization expect Unicode systems to do things it cannot do by just replacing strings. You indeed frequently run into issues even in European languages if you just think it's a matter of replacing strings.

Re:The ü/ü Conundrum by earthboundkid on Friday March 29, 2024 @05:41PM (#39869322)
> It's not a myth, as anyone living in Japan knows

I lived in Japan. It is a myth. :-¥