News for hackers. Stuff that matters.
Re:Make JDK source code UTF-8 by astrange on Tuesday February 28, 2023 @11:38PM (#34978830)
There's no difference between encoding a page in Shift_JIS and tagging the page as Japanese language (assuming your browser switches fonts correctly based on language tags). Chinese text in Shift_JIS isn't going to look correct either.
Re:Make JDK source code UTF-8 by lmm on Tuesday February 28, 2023 @11:44PM (#34978860)
Sure, but every application handles encoding (or at least, every application did up until the recent UTF-8-only movement), whereas language tagging is web/html-specific.
Re:Make JDK source code UTF-8 by astrange on Wednesday March 01, 2023 @12:02AM (#34978971)
If you're really hardcore processing multilingual text you need an equivalent to language tags anyway, because you need a dictionary for word-wrapping, date formatting, quote marks etc along with changing fonts (or glyph selectors in the same font.) TeX has them too.

But usually people only care about their language, so it goes by the system UI language and it works out.

Re:Make JDK source code UTF-8 by lmm on Wednesday March 01, 2023 @05:21PM (#34989162)
> If you're really hardcore processing multilingual text you need an equivalent to language tags anyway, because you need a dictionary for word-wrapping, date formatting, quote marks etc along with changing fonts (or glyph selectors in the same font.) TeX has them too.

Sure. And once you're doing that you don't gain a lot of benefit from unicode AFAICS, because you have to track these spans of locale-specific text.

Re:Make JDK source code UTF-8 by astrange on Wednesday March 01, 2023 @06:54PM (#34990073)
Lets you copy characters that are the same (like people's names) between text though.
Re:Make JDK source code UTF-8 by lmm on Wednesday March 01, 2023 @08:05PM (#34990618)
People's names are the thing you should be most careful to not do that with! Most Japanese people will generally put up with seeing the Chinese rendering of a character in ordinary text, but they really don't want you to do that to their names!
Re:Make JDK source code UTF-8 by TheDong on Wednesday March 01, 2023 @01:24AM (#34979359)
I think the suggested alternative though is having UTF-8 take care of this.

UTF-8 already has different codepoints for every other language, like I can type russian (День), greek (Ημέρα), english, even egyption hieroglyphs (𓀃) all of them in this one text field, and they all render right for both of us if we have a suitable font. If they don't render, it's unambiguous.

It's only chinese and japanese where I can type some characters, and depending on if you have a chinese or japanese font _first_ in your system, it might render wrong, and either way it's ambiguous to the computer without further metadata. The computer doesn't need metadata to know that "α" is "α", why does it need metadata to know if 直 is the chinese (http://www.hanzi5.com/bishun/76f4.html) or japanese (https://kanjivg.tagaini.net/viewer.html?kanji=%E7%9B%B4) character, two different looking characters that share the same codepoint.

Re:Make JDK source code UTF-8 by yorwba on Wednesday March 01, 2023 @03:55AM (#34980213)
Unicode does not have different codepoints for every language. Leaving aside scripts that aren't encoded at all, all languages using the Latin script get a single set of glyphs, and if you want Comic Sans MS or Carolingian miniscule or fine-grained control over whether "a" has a hook at the top or not, you need to specify a font that renders it the way you want, just like with 直.

Because Unicode is about the meaning of characters, not their appearance. If there were a context where the two different shapes of 直 have different meaning, Unicode would add a new codepoint to distinguish them. In fact "ɑ" without hook does have a separate codepoint because it is used in linguistics to mark a vowel different from "a".

Re:Make JDK source code UTF-8 by lozenge on Wednesday March 01, 2023 @04:23AM (#34980356)
If CYRILLIC SMALL LETTER A deserves a code point despite being so similar to LATIN SMALL LETTER A, why doesn't a Japanese character that actually looks different from a Chinese one get a codepoint? And the idea that something can have the "same meaning" across two languages is very wooly.
Re:Make JDK source code UTF-8 by astrange on Wednesday March 01, 2023 @06:53PM (#34990063)
Why don't Japanese characters in seal script or handwriting get different codepoints from mincho font characters? They look different too.