It'll be interesting to see how LLMs do when trained on concepts (ie the Chinese alphabet), rather than sounds, though.
From my understanding (which only comes from reading HN), unique token count isn't an issue that LLMs run into. If it is, then that would be a bummer for the possibilities of exploiting all the features of the Chinese language.
> and if you use, say, pinyin
Well yeah pinyin would be inefficient because you're stripping away all the "built in" semantics of a Chinese character to only focus on the pronounciation.
Granted, that doesn't apply to Chinese, which encoders concepts, so that's interesting to see in LLMs.
I've learned languages with alphabets and glyph scripts like Chinese, and *what I don't like about alphabet languages like English is that the letters themselves provide little context to their meaning, although you could learn the latin root words and guess from there. Of course you know how to say it, but you don't know what it means. With Chinese, you might guess what it means, but you don't know how to say it.* The characters have more meaning in them, although this isn't the case for every word, which is something non-speakers assume.
In short, from my experience learning Chinese was way easier than learning Vietnamese, which have historical ties together since Vietnamese used a fork of Chinese script until the early 1900s when they transitioned to latin script in order to improve literacy rates. Sure, it improved literacy rates because to read you only need to know how to pronounce, but it doesn't mean glyph languages are harder to learn the meaning/semantics.
Look up Byte-Pair Encoding (BPE) for a fascinating and simple algorithm that explains the process of choosing the tokens. It shows why it doesn't matter how token dense the source language is.
Also, in case of english language, it's one token per one word (more or less), so it's the same as in Chinese - assuming both LLM tokenisers were geared towards their native language.
The only issue is when you have an tokeniser geared towards western languages, and you try to to use the same tokeniser on a different group of languages - then a single word in a language foreign to the tokeniser would have multiple tokens.
But that has nothing to do with the underlying structure of the language.
In other words - you wouldn't really see a difference between an input in chinese compared to english after the text gets tokenised. It's rougly the same amount of tokens, and the underlying parameter count would be also similar.
I think it's easier to connect semantic meaning between these tokens/charcaters, because you can relate a character to other characters by their presence in words (most words are 2 characters long (compound words), although there are sometimes words like AB and BC which mean the same thing, but in conversations people will just say A or C and the meaning (AB or BC) is inferred by the context of the conversation, so in a semantic meaning database you would want a connection between A and AB, and to BC & C). You could also have a dimension for each radical that isn't a phonetic component. Radicals do link vaguely related words together, and I think this is a powerful language feature missing in English that could be exploited in the NLP field.
I'm not a fluent or proficient speaker, so maybe I'm not the best person to explain things, but before OpenAI's Whisper, speech-to-text in WeChat in Mandarin was way better than any English speech-to-text I've ever used. The structure of the Chinese language and its large userbase would make me bet long on Chinese NLP progress over the west.