Re:Unicode character “ꙮ” (U+A66E) is being updated
by lmm on Monday September 19, 2022 @06:29PM
(#32904997)
Meanwhile one still can't roundtrip regular Japanese without some kind of funky out-of-band signalling. By itself this kind of thing is harmless, but it speaks to poor prioritization from Unicode.
[ Parent
| on HN ]
Re:Unicode character “ꙮ” (U+A66E) is being updated
by lifthrasiir on Monday September 19, 2022 @08:38PM
(#32906124)
This is incorrect. I think you defined round-trip as something else, but some character set A providing a round-trip compatibility with other set B means that B can be converted to A and back to B without a loss. And it is one of Unicode's explicit goals to provide a round-trip compatibility with major encodings including Japanese ones.
Han unification only means that when you convert Japanese encodings (B) to Unicode (A), it is not distinguishable from non-Japanese encodings converted to Unicode. This means that the Unicode text doesn't always follow domestic conventions without out-of-band signaling or IVD or so. But if you know that the text was converted from a particular encoding, you can perfectly recover the original text encoded in that encoding.
Re:Unicode character “ꙮ” (U+A66E) is being updated
by lmm on Monday September 19, 2022 @11:00PM
(#32907157)
By that logic any 8-byte encoding is round-trip compatible with all encodings, since however bad the mojibake is, if you know what the original encoding was then you can always just convert back to that.
Re:Unicode character “ꙮ” (U+A66E) is being updated
by lifthrasiir on Monday September 19, 2022 @11:43PM
(#32907418)
Yes, but only under a very wrecked hypothetical definition of "conversion".
Re:Unicode character “ꙮ” (U+A66E) is being updated
by josefx on Tuesday September 20, 2022 @06:06AM
(#32909812)
To be fair they wanted to keep everything representable with 16 bits and that wasn't going to happen without the Han-unification. The mess when everything still had to move to a 32 bit representation has been far reaching, many programming languages went from exposing code points atomically as "char" to some half encoded nonsense value that just happens to also be a valid standalone value in UTF-16 most of the time and a source of bugs when you least expect it.
Re:Unicode character “ꙮ” (U+A66E) is being updated
by umanwizard on Monday September 19, 2022 @07:56PM
(#32905793)
Why can’t it round-trip Japanese?
Re:Unicode character “ꙮ” (U+A66E) is being updated
by lmm on Monday September 19, 2022 @08:13PM
(#32905930)
"Han Unification" - in Unicode many Japanese characters are represented as Chinese characters that look different (and subjectively ugly). The Unicode consortium's answer is that you're supposed to use a different font or something when displaying Japanese, which is pretty unsatisfying (e.g. if you want to have a block of text that contains both Japanese and Chinese, you can't represent that as just a Unicode string, it has to be some kind of rope of segments with their own fonts, at which point frankly you might as well just go back to bytes-with-encoding which at least breaks very clearly and visibly if you get it wrong).
Re:Unicode character “ꙮ” (U+A66E) is being updated
by kevin_thibedeau on Monday September 19, 2022 @09:24PM
(#32906497)
You can use the deprecated language tag control codes to distinguish unified code points. It is unlikely to be well supported but it is there.