Re:Generative AI uses more energy on non-English text
by kgeist on Saturday November 14, 2026 @10:34PM
(#50503741)
There's Russian research on this, they swapped the tokenizer of an existing model (IIRC something from the Qwen family) for one which had far more Cyrillic tokens. After some post-training, the model kept its original quality, but it used way fewer tokens.
Seems a bit dated. Sarvam has been taking about the inefficiency of the tokenization schemes used by mainstream models when it comes to Indic-languages for years now.