I feel like that info is placed appropriately. The abstract mentions that energy disparity persists across models, the rest follows the standard Intro -> Related Work -> Experiments structure

Is the publishing cycle this long? Those models were relevant 18 months ago

Working out the kinks in your methodology obviously takes some time. But is there any structural reason not to rerun the benchmarks on current models in the final stages before submission? Assuming the results generalize to those newer models

Unfortunately it is this long. I know people in academia studying human-AI interfaces and up until recently they were still using gpt-3.5.

Yes and let's all remember that lag when the studies about AI outcomes are posted.

Why? Seems like an appropriate counterbalance to be insane promises and moon shots that CEOs are promising

It's challenging to keep running experiments. In a perfect world, we would run our experiments again and update the results. But some experiments are expensive, and some experiments are inherently difficult to reproduce. I have a paper nearing completion that involves interaction between trained (human) professionals and a model. It was a logistical challenge to bring those people together (also, we had to pay them), and the roleplaying that we asked LLMs to do can't be done anymore--they refuse for safety reasons. Stuff like this is pretty common.

I am hopeful that we can eventually move to open weights models, but at the time we did the experiments, the open weights models did not have the capabilities we needed. The ever-shifting sands of frontier models makes reproducibility difficult, and not just for scientists.

> Is the publishing cycle this long?

I did a PhD and the answer is yes. Often longer than this.

How would one evaluate the energy consumption of models accessible only through Anthropic or OAI APIs? You can't exactly fire them up on your own servers and measure power consumption.

It's explained too..

Look at the token usage counts.

The Anthropic and OAI APIs report token usage (input, output, and in some cases reasoning/thinking). The input tokens consist of the system prompt, the user input/messages, and any other context data.

Given the same model the only variable is the token count. The more tokens, the more passes through the LLM (to generate the next output token), the more energy used.

It's well known that non-English text consumes more tokens [1], [2] and thus more energy. This is due to two factors:

1. Latin script languages having different character pairings to English and modified characters, including language features like modifying verbs for person/tense/etc. (e.g. Spanish);

2. Non-Latin scripts that take more bytes to represent in UTF-8 and (esp. for ideographic and syllabic languages like Chinese, Japanese, and Korean) have lower common character pair sequences, meaning shorter tokens for those scripts and thus more tokens for a given text sequence.

[1] https://ikriv.com/blog/?p=5322 "LLM tokens and foreign languages" (5 Aug 2025)

[2] https://medium.com/@programmerraja/how-your-llm-costs-5x-mor...

So basically, the tokenizer is optimized for English and performs worse on other languages?

roughly the tokenizer is optimized such that the training data (for the tokenizer) can be represented with the smallest number of tokens. and yes, English tends to be overrepresented in training data.

> token usage

It is not exactly accurate, because it ignores speculative decoding which can often save a lot of time, compute and energy.

See this branch: https://news.ycombinator.com/item?id=50441444#50497854

Speculative decoding will be model/software based and apply to all languages. The time, compute, and energy will be less compared to other models, but for the same model, hardware, and software used my reasoning still applies -- comparing English to other languages (unless optimized by the tokenizer) then English will consume less tokens and thus less energy. That is, it is still a function of the tokens used.

Your counterpoint is incorrect. The savings of speculative decoding are a direct function of how hard it is to predict the next token. Which means that for a specific task, the smaller model can be used more in a language which uses more tokens to say the same thing (to perform the same task), as the difficulty of the task is fixed between languages and the language that takes more tokens "spreads the same difficulty" over more tokens.

You may be right, but I'd need to see the data and analysis of the number of passes through each stage for the various languages.

The reason I think speculative decoding would scale the same is that the smaller predictive model has the same token behaviour as any other model compared to the other models. So the only savings would be if the smaller model can better predict more tokens for non-English compared with English such that the ratio of tokens between the two becomes the same or lower for the non-English language.

For Latin-script languages like Spanish I can see the smaller model predicting verb endings correctly, improving performance there. These languages have 1.5x-2x more tokens [1] so will need to be 1.5x-2x better than English in the speculative decoder for the performance of the non-English languages to be comparable.

For Han characters (Chinese and Japanese) it looks like the tokenizer is splitting the characters in the middle of UTF-8 byte sequences, at least in [1] where the number of tokens is larger than the number of Han characters. This would potentially make the paired tokens harder to predict by the smaller model. Given that there are ~2 Han tokens per English token the speculative decoding would need to be 2x better for Han than for English to make it perform the same compute energy wise.

If we assume that the smaller decoder model is perfect and that there is no additional cost on the verifier then the English vs non-English languages are going to scale the same on the smaller model, so speculative decoding would just be like using that smaller model and the non-English languages would perform worse due to them having more tokens that need to be processed by the smaller model.

The other factor is the context length of the transformers in the verifier. The question then is how quickly does that fill up and how many passes through that are needed for English (fewer tokens) compared to non-English (more tokens).

[1] https://ikriv.com/blog/?p=5322

You say this is well-known, but my experience is talking chinese to the chinese LLM produces more efficient work, with less rambling in the thinking/reporting steps, and I have been presuming that it is because longer utf8-token still beats a bunch of utf7-compatible words for expressing the kinds of big ideas maths and code requires. 省字臃思

So I have to wonder why is token count so high in their examples? Here is my theory: I would not say 我遇见了一只大狗, but I would just say 遇大狗. I know I can't do that in English, but I think surely the researchers should know (or would ask someone?) how this works trying to compare phrases in isolation, so I wonder why is this test meaningful to them? Why are they doing that? Are they looking for this result? 字省思臊 I don't talk like google translate.

So who are these people and why are you listening to them and repeating what they say like it is so? Why exactly are you so convinced?