News for hackers. Stuff that matters.
Re:Generative AI uses more energy on non-English text by lifthrasiir on Saturday November 14, 2026 @05:41PM (#50501758)
I've done my own research on that as a part of my Aqueous [1] archiver, and the conclusion is that the inherent information entropy is about as same (CJK scripts, for example, compresses much better when they are turned into their native character encodings). So a script-aware tokenization should make that irrelevant.

[1] https://lifthrasiir.github.io/aqueous/ (specifically wrt-recons-hangul codec)

Re:Generative AI uses more energy on non-English text by stephantul on Sunday November 15, 2026 @04:45AM (#50505490)
The models the top post refers to are not script aware. This is my point