News for hackers. Stuff that matters.
Re:Generative AI uses more energy on non-English text by kgeist on Saturday November 14, 2026 @10:34PM (#50503741)
There's Russian research on this, they swapped the tokenizer of an existing model (IIRC something from the Qwen family) for one which had far more Cyrillic tokens. After some post-training, the model kept its original quality, but it used way fewer tokens.

There are no comments on this comment yet.