That's surprisingly energy efficient, for a technology that seemed like magic back in 2022.

How in the world do they claim to have 0.24Wh per query? MIT estimates the LLama 3.1 405B model uses 1.86Wh.

GPT-4 is over a trillion parameters. Is there any reason to think they have 2.5x more parameters but somehow use 6x less energy?

Mixture of Experts. Llama 3.1 405B is a dense model, so to evaluate the next token, the context has to go through literally every parameter in its neural network. Whereas with mixture of experts, it's usually like a sixth to a tenth or even less of the neural network parameters that actually get evaluated for every token. Also, they don't use GPT-4 or 4.5 anymore iirc, which may have been dense (and that's why they were so expensive), 4.1 and 4o are much different models.

The median prompt being a google search, not agentic coding for example.

I also think this misses a bit the forest from the trees, emissions need to go down across all industries and tech is actively increasing theirs.