News for hackers. Stuff that matters.

A comment on Generative AI uses more energy on non-English text. Parent comment

Re:Generative AI uses more energy on non-English text by brainwad on Saturday November 14, 2026 @12:47PM (#50499073)
It's not optimised deliberately for English, right? It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Even if you use bytes only, Unicode is biased towards English and other Latin languages, too.

[ Parent | on HN ]
Re:Generative AI uses more energy on non-English text by cubefox on Saturday November 14, 2026 @12:55PM (#50499159)
They are typically optimized for English, for example, they often use rules for English contractions. See this thread: https://x.com/_albertgu/status/2113687798826111283

I don't know whether European languages work inherently better with tokenizers, but I don't see why they would.

[ Parent | on HN ]
Re:Generative AI uses more energy on non-English text by sznio on Saturday November 14, 2026 @12:56PM (#50499162)
logographic languages still win I guess, a 4-byte chinese character beats a n-character English word for that same concept.

but yeah, anything alphabetic yet not latin gets the worst tradeoff.

[ Parent | on HN ]
Re:Generative AI uses more energy on non-English text by brainwad on Saturday November 14, 2026 @01:23PM (#50499403)
Most CJK characters are 3 bytes in UTF-8. But then most English words translate to more than one character, so 5 bytes might be more realistic (apparently Chinese averages 1.6-1.7 characters per word).
[ Parent | on HN ]
Re:Generative AI uses more energy on non-English text by vova_hn2 on Saturday November 14, 2026 @01:02PM (#50499210)
> It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Not really, because I don't think anyone use the whole training dataset to train the tokenizer.

From my brief (disclosure: claude-assisted [0]) internet search I learned that typically they use from tens to hundreds GB of data to train the tokenizer. Sometimes it is hand-curated, sometimes it is randomly sampled.

Of course, you can deliberately choose how much text in each language do you include in the training set for the BPE and deliberately optimize for (or against) a specific language.

[0] https://claude.ai/share/da277189-f23a-4071-bd9f-2d75fe6aa779

[ Parent | on HN ]
Re:Generative AI uses more energy on non-English text by Phemist on Saturday November 14, 2026 @01:04PM (#50499228)
Ive always thought that Serbian is the worst off out of all European languages. Both cyrillic and latin script is widely used. Latin also has a naked variant, where all diacritics are left out (which is especially prevalent online). Also, it is non-EU, so initiatives like [0] will only indirectly help with representation (croatian being the same language, but it does not use cyrillic script)

[0] https://digital-strategy.ec.europa.eu/en/news/commission-wel...

[ Parent | on HN ]
Re:Generative AI uses more energy on non-English text by numpad0 on Saturday November 14, 2026 @02:41PM (#50500083)
[delayed]
[ Parent | on HN ]
Re:Generative AI uses more energy on non-English text by brainwad on Saturday November 14, 2026 @04:08PM (#50500836)
It is desirable for all alphabets/abjads/abugidas/syllabaries, not just Latin. Chinese is rather the special case of a human language with very high character inventory and hence low character/word ratio.
[ Parent | on HN ]