News for hackers. Stuff that matters.
Re:LLM companies to access largest Chinese non-fiction book ... by visarga on Sunday November 05, 2023 @05:03AM (#38149743)
Text tokens by their construction are more like the Chinese ideographic characters, so no, there would not be a big difference between tokenized English and Chinese. The number of unique tokens is decided upfront - something like 50K - so, much more than basic vocabulary. English tokens can even be 2 word phrases.

Look up Byte-Pair Encoding (BPE) for a fascinating and simple algorithm that explains the process of choosing the tokens. It shows why it doesn't matter how token dense the source language is.

There are no comments on this comment yet.