Generative AI uses more energy on non-English text

Re:Generative AI uses more energy on non-English text by gavinsyancey on Saturday November 14, 2026 @11:57AM (#50498538)

So basically, the tokenizer is optimized for English and performs worse on other languages?

[ Parent | on HN ]

Re:Generative AI uses more energy on non-English text by dosinga on Saturday November 14, 2026 @06:29PM (#50502195)

roughly the tokenizer is optimized such that the training data (for the tokenizer) can be represented with the smallest number of tokens. and yes, English tends to be overrepresented in training data.

[ Parent | on HN ]