My understanding is that it’s about tokenization. English input is converted into less tokens because tokenizer is optimized for English. This makes huge difference because all the output tokens are fed back into the input at each token generation. And attention blocks are generally quadratic with the input tokens. So foreign language means more tokens everywhere and more tokens mean more passes and much more compute and memory.

This is absolutely correct.

But I would subtly add it's also about the training data, which is optimized towards English. Not intentionally, just incidentally. (And this is a really hot area of improvement with a long way to go.)

Which is largely why the tokenization part that you mentioned is what it is.

My understanding (which may be wrong) is that English is best but most of the Romance languages do okay because English has borrowed from them so much.

I think you only start getting seriously bad impacts in non-Romance languages where they have their own alphabet, or use the English alphabet phonetically causing letters to appear next to each other that never would in English. E.g. in English "qu" is a pretty obvious token, you rarely see a Q without a U after it. Happens all the time in anglicized Middle Eastern words, though. Quran is like a whole other thing, you'd almost always expect another vowel after qu, not a consonant.

I've heard English is one of the most expressive languages, but is complex to learn due to absorbing words and ideas from so many other languages. Lots of contradictions, situational rules, and unwritten rules (like the order of adjectives).

Given the complexities involved, maybe we should be impressed by someone who masters even one!

Can't you train a model to translate tokens from human languages to a separate NN token language with a compressed representation like a VAE? It seems like a leaky abstraction to have these trillion parameter models be dependant on the fact that English gets tokenized in a certain way, and I assume this has been researched a lot already.

Having studied Japanese at university and living there for a bit my interest naturally flowed in that direction, so I ran a quick check on OpenAI's tokeniser, nothing close to an actual study. I ran two tests of everyday English and its Japanese equivalent and another using more technical vocabulary across both languages. These were my results: Tokeniser Everyday JP/EN Technical JP/EN GPT-3 2.69x (183/68) 3.48x (233/67) GPT-4 2.16x (147/68) 2.53x (167/66) GPT-5.x 1.49x (101/68) 1.74x (115/66) It shows that the gap clearly has shrunk a fair bit with Japanese going from 2.7x English on GPT-3 to 1.5x on GPT-5.x for everyday text. The same passage of text yielded flat results for English across each tokeniser, so all the improvement went into Japanese. I also found that the Japanese polite form used to cost extra on GPT-3 but is now effectively free. The polite version of the Japanese passage was 9 characters longer (133 vs 124), and contained endings such as ました, ません and でした which have all become single tokens in newer tokenisers. Perhaps most interestingly, katakana loanwords seemed to add a lot of extra cost. For example, a word like パスワード(password) is split into four tokens (パ / ス / ワ / ード), while "password" is one token in English. In this example, the exact katakana string likely did not appear frequently enough in the training data, unlike the common hiragana clusters such as the polite endings, and the English word “password”, which earned their own tokens. The newer tokenisers continue to split less common kanji (like 靴 and 濡) into byte pieces, but katakana stands out as even everyday words are broken apart. It'd be interesting to run these checks on the same Qwen, Gemma, Llama tokenisers the study uses.

"Reason about it" is where your abstraction has sprung a leak.

LLMs do what they say on the tin: they model language. If you ask an LLM "find security vulnerabilities in my code" for hello_world.rs, the next likely tokens are "I found the following vulnerabilities", regardless of what logic or reason say.

It happens to be true that predicting next tokens on your own output mocks actual reasoning fairly well. But it still just a simulacrum rather than the real thing, and when held as an abstraction it leaks everywhere.

Understanding this is critical to avoiding the common failure case of scope creep and code slop.

There’s enough experience with LLMs now and comparisons to our own thinking that we can say LLMs are thinking and doing so not so differently from us - at least for some aspects of reasoning.

Trying to pretend otherwise at this point doesn’t help us to utilize them or understand their limits. We’re better off accepting that intelligence isn’t unique to us as a species. Then we can see that modeling language as a tool for us as a vehicle for thinking and reasoning, which philosophers like Wittgenstein already landed upon before LLMs.

I only used "reason about" as shorthand for the computation that takes place between input text gets tokenized and output tokens gets translated back to text. There are a lot of layers in the middle where the tokens no longer represent text directly but rather a combination of the different tokens through the attention and MLP projections.

No decent modern LLM is going to start its reply with “I found the following vulnerabilities” when you ask it to find security vulnerabilities in a straightforward hello world.