News for hackers. Stuff that matters.
Re:You Should Probably Pay Attention to Tokenizers by sundarurfriend on Wednesday October 23, 2024 @05:43PM (#41929527)
> Besides that, for alphabetic languages, there exists almost no relation between form and meaning.

Also true for Abugida-based languages, for eg. சரம் (saram = string) vs மரம் (maram = tree), and many more. I think your intention with specifying "alphabetic languages" was to say "non-logographic languages", right?

Re:You Should Probably Pay Attention to Tokenizers by bunderbunder on Wednesday October 23, 2024 @06:08PM (#41929677)
I'll do you one more and say "non-Chinese languages". Written Japanese - including the kanji portion of the script - has the same characteristic.

And even in Chinese it's a fairly weak relationship. A large portion of the meanings of individual characters come from sound loan. For example the 英 in 英雄 means "hero", in 英语 means "England", an in 精英 means "flower". The relationship there is simple homophony.

On the other hand, one thing you do get with written Chinese is that "1 character = 1 morpheme" very nearly works. So mechanistically breaking a text into a sequence of morphemes can be done pretty reliably without the aid of a semantic model or exhaustive hard-coded mapping. I think that for many other languages you can't even get close using only syntactic analysis.

Re:You Should Probably Pay Attention to Tokenizers by thaumasiotes on Wednesday October 23, 2024 @08:55PM (#41930827)
> I'll do you one more and say "non-Chinese languages". Written Japanese - including the kanji portion of the script - has the same characteristic.

Written Japanese is much more ideographic than written Chinese. Japanese spelling is determined, such as it is, by semantics. Chinese spelling is determined by sound. Thus, 女的, 娘们, and 妮子, all meaning 'girl' or 'woman', have no spelling in common because they are different words, while Japanese uses 女 for "jo" and "onna" despite a total lack of any relationship between those words.

Re:You Should Probably Pay Attention to Tokenizers by bunderbunder on Thursday October 24, 2024 @11:11AM (#41936303)
Chinese spelling is also determined by meaning. 是、事、and 式 are all pronounced shì. Which one you use depends on whether you mean the copula, an event/business/matter, or a test.

I see that your examples are getting at how character components work in Chinese, but it's oversimplistic. 女(nǚ) means "woman", but 娘 (niáng) means "mother". It's composed of the functional components 女, which is just a hint at the meaning for disambiguation purposes, and 良 (liáng), which is a hint at the pronunciation. Similar for 妮 (nī) with sound component 尼 (ní) and a similar historical meaning and development to the English word "lass". In the second and third cases the 女 is a semantic functional component, but it's still a different character with a different (if similar) meaning.

re: What I said about Kanji; I don't speak the language so perhaps I misunderstand how things work. My impression was that the mapping from character to meaning was looser in Japanese than it is in Chinese.

Re:You Should Probably Pay Attention to Tokenizers by stephantul on Thursday October 24, 2024 @12:05AM (#41931796)
I was trying to say “at least for alphabetic languages”. I don’t like to say things about languages I can’t speak or write. So, no, it wasn’t my intention to say “non-logographic languages”