Re:Generative AI uses more energy on non-English text
by rhdunn on Sunday November 15, 2026 @03:35AM
(#50505143)
You may be right, but I'd need to see the data and analysis of the number of passes through each stage for the various languages.
The reason I think speculative decoding would scale the same is that the smaller predictive model has the same token behaviour as any other model compared to the other models. So the only savings would be if the smaller model can better predict more tokens for non-English compared with English such that the ratio of tokens between the two becomes the same or lower for the non-English language.
For Latin-script languages like Spanish I can see the smaller model predicting verb endings correctly, improving performance there. These languages have 1.5x-2x more tokens [1] so will need to be 1.5x-2x better than English in the speculative decoder for the performance of the non-English languages to be comparable.
For Han characters (Chinese and Japanese) it looks like the tokenizer is splitting the characters in the middle of UTF-8 byte sequences, at least in [1] where the number of tokens is larger than the number of Han characters. This would potentially make the paired tokens harder to predict by the smaller model. Given that there are ~2 Han tokens per English token the speculative decoding would need to be 2x better for Han than for English to make it perform the same compute energy wise.
If we assume that the smaller decoder model is perfect and that there is no additional cost on the verifier then the English vs non-English languages are going to scale the same on the smaller model, so speculative decoding would just be like using that smaller model and the non-English languages would perform worse due to them having more tokens that need to be processed by the smaller model.
The other factor is the context length of the transformers in the verifier. The question then is how quickly does that fill up and how many passes through that are needed for English (fewer tokens) compared to non-English (more tokens).
The reason I think speculative decoding would scale the same is that the smaller predictive model has the same token behaviour as any other model compared to the other models. So the only savings would be if the smaller model can better predict more tokens for non-English compared with English such that the ratio of tokens between the two becomes the same or lower for the non-English language.
For Latin-script languages like Spanish I can see the smaller model predicting verb endings correctly, improving performance there. These languages have 1.5x-2x more tokens [1] so will need to be 1.5x-2x better than English in the speculative decoder for the performance of the non-English languages to be comparable.
For Han characters (Chinese and Japanese) it looks like the tokenizer is splitting the characters in the middle of UTF-8 byte sequences, at least in [1] where the number of tokens is larger than the number of Han characters. This would potentially make the paired tokens harder to predict by the smaller model. Given that there are ~2 Han tokens per English token the speculative decoding would need to be 2x better for Han than for English to make it perform the same compute energy wise.
If we assume that the smaller decoder model is perfect and that there is no additional cost on the verifier then the English vs non-English languages are going to scale the same on the smaller model, so speculative decoding would just be like using that smaller model and the non-English languages would perform worse due to them having more tokens that need to be processed by the smaller model.
The other factor is the context length of the transformers in the verifier. The question then is how quickly does that fill up and how many passes through that are needed for English (fewer tokens) compared to non-English (more tokens).
[1] https://ikriv.com/blog/?p=5322