Re:Generative AI uses more energy on non-English text
by lifthrasiir on Saturday November 14, 2026 @05:41PM
(#50501758)
I've done my own research on that as a part of my Aqueous [1] archiver, and the conclusion is that the inherent information entropy is about as same (CJK scripts, for example, compresses much better when they are turned into their native character encodings). So a script-aware tokenization should make that irrelevant.
[1] https://lifthrasiir.github.io/aqueous/ (specifically wrt-recons-hangul codec)