News for hackers. Stuff that matters.
Re:You Should Probably Pay Attention to Tokenizers by thaumasiotes on Wednesday October 23, 2024 @08:55PM (#41930827)
> I'll do you one more and say "non-Chinese languages". Written Japanese - including the kanji portion of the script - has the same characteristic.

Written Japanese is much more ideographic than written Chinese. Japanese spelling is determined, such as it is, by semantics. Chinese spelling is determined by sound. Thus, 女的, 娘们, and 妮子, all meaning 'girl' or 'woman', have no spelling in common because they are different words, while Japanese uses 女 for "jo" and "onna" despite a total lack of any relationship between those words.

Re:You Should Probably Pay Attention to Tokenizers by bunderbunder on Thursday October 24, 2024 @11:11AM (#41936303)
Chinese spelling is also determined by meaning. 是、事、and 式 are all pronounced shì. Which one you use depends on whether you mean the copula, an event/business/matter, or a test.

I see that your examples are getting at how character components work in Chinese, but it's oversimplistic. 女(nǚ) means "woman", but 娘 (niáng) means "mother". It's composed of the functional components 女, which is just a hint at the meaning for disambiguation purposes, and 良 (liáng), which is a hint at the pronunciation. Similar for 妮 (nī) with sound component 尼 (ní) and a similar historical meaning and development to the English word "lass". In the second and third cases the 女 is a semantic functional component, but it's still a different character with a different (if similar) meaning.

re: What I said about Kanji; I don't speak the language so perhaps I misunderstand how things work. My impression was that the mapping from character to meaning was looser in Japanese than it is in Chinese.