Which is interesting, given that life itself is merely that. It's just that humans decided that that ain't good enough and invented something alien to nature - logic.
Hm. I would actually go so far as to say "predict" is just the wrong word to use in 2026, as it drags in a picture of what LLMs are that's from the pre RLVR era, and which is frequently used to suggest they have all sorts of limitations that they've now surpassed ("an LLM will never solve a major open mathematics problem, they're just predicting tokens"). These days I tend to use the more neutral phrase "generate the next token"; I'd only say "predict" in the technical context of (say) the pre-training part of the training run where the object is indeed to minimize the cross-entropy loss function by predicting next tokens in existing text. Generating a token with a modern LLM is better described as something like, "given the totality of its training, including not just next token prediction with pre-training but also solving vast numbers of math and coding problems with RLVR, the LLM generates tokens one by one."
Anyhow. I'm not sure the weight understanding is all that strange. (It's impressive, but so are many thing LLMs do these days.) LLMs are very very good at coding and math problems, of which this is one, and in particular, their RLVR training presumably includes many tasks involving neural networks. It's amazing and impressive, I agree, but I'd say less amazing than proving major open theorems in mathematics, which we already know they can do.
Substitute ‘generate’ and the meaning is unchanged.
It's worth reflecting on the original domain jargon for selecting the next token: completion.
And yes, I understand "prediction" is a standard term in the field. I just think it is a bad one and we should stop using it.
This doesn't make it any less fascinating that by combining this basic "do as I say" training with massive amounts of all of humanity's data, you can solve major problems—that is still incredible. But that doesn't fundamentally change what is going on, mechanistically.
I don't understand these pseudo-religious desires to read into models something "more" than what we know. We built them. We don't go around making wild claims about how cars are actually more than just a mechanical system, how they aren't just taking us places—we do, but usually such points are cultural not about the mechanica capabilities of cars. People in the LLM space seem to be constantly conflating these two views. It's entirely accurate to say, on the one hand, next token predictors combined with RL, and massive troves of existing human work, can solve culturally interesting problems, and to hold, on the other hand, that mechanistically what is happening is statistical next token predictions over global linguistic space and more or less "monkey see monkey do".
Mathematics models the world, it does not have substance. It is a useful way of looking and a useful way of manipulating, but many models are equally useful. Capital T Truth, beyond models is very difficult to get at. We can show that things are true for certain models well enough. Showing that something is true in all models (a universal truth) is far more difficult. You face incommensurability.
I see a lot of people in technology fall into this extremely basic logical fallacy called Reification constantly.
I mean, when you really get down to it, either the particles and such operate on very specific rules, and the rules govern how all our brain and body work. Followed to the conclusion, the universe is computable and we can simulate life.
You can demonstrate definitively that there nothing other than "next token prediction" by running the same inference engine/model/prompt/settings multiple times in a row on the same hardware at temp 0 with a seed set.
If you turn the temp up, then you might let the "mechanism" select pseudorandomly from a probability distribution of k-likely tokens with minimum p-cumulative probability. Even then, if the next token has a very high probability, deviation will be uncommon.
I just performed this experiment right now on a few dozen runs of a dense model at thousands of tokens of output at temp 0, and all were identical. Then I did it with a MOE model, which might get some benefit of picking different experts each time. Nope. Outputs were again identical.
Describing LLMs as next token predictors isn’t incorrect, it’s meaningless. It’s the equivalent of describing brains as “a bunch of atoms”.
The problem with these descriptions is that they say nothing about high-level capabilities. If you can predict the next token in a proof of the Riemann hypothesis, then you can prove the Riemann hypothesis.
The part that really bakes my noodle: `/foo/bar/baz.txt` can be a filename that no one has ever typed before, the error inside the file can be a typo that no one has ever typed before, and it will still correctly "predict the next token" to solve your problem.
In the case of the brain, it'd be more akin to the reduction of the brain to just the neuronal dynamics (obviously experience phenomenally is much more than that to us!)
But both views can be useful, depending on what you are trying to understand.
The problem with these arguments is that people pick and choose based on their persons preferences. People who are in love with LLMs will, on the LLM side, argue against reductionism "they are way more than next token predictors!" then they'll turn around and equally try to argue "the substrate doesn't matter! The brain is "just" neurons and digital activity so of course we can replicate it!" (this is also incorrect, btw, the brain is also an analog system, even in its electrical connections and not just in its chemistry)
The on the other side the anti LLM crew will do the opposite (launch into reductionism when it comes to LLMs, resist it when it comes to humans). I'm a human being and I like humans and I think people have adopted a strange view of LLMs, which are a tool and should be treated as such, so I tend to side with the humans. The stalwart defenders of LLM honor that would balk at any attempt to reduce them in stature to what they fundamentally are, mechanistically, are... an odd bunch. I'm confused why people are so passionate about it. If someone told me "well your IDE is just a fancy AST walker" yeah I think that is a dumb take but I wouldn't feel like I had to argue against such a take either saying "no! no! it's so much more!"
A system is not just its little bits.
Where "system" and "little bits" can be just about anything. But people endlessly repeat this mistake, and it has been a go-to misunderstanding/misrepresentation for a lot of people on this forum.
if it was designed to edit code, id expect the first translation from latent space out to tokens to be the final code, not be assembling the code by random number generator.
its not really designed to edit code until the tests succeed, we try to manipulate the distribution of tokens to predict by prompt engineering with the test results.
Going back to first principles, wikipedia mentions [0] inference as:
> "Inferences are steps in logical reasoning, moving from premises to logical consequences. Inference is traditionally divided into deduction and induction"
Is that exactly what next token predictor means?
Training is not design. Design is the mechanism, namely a statistical method of learning distributions, and of executing those distributions computationally as an IO function.
The model is not designed to write unit tests. It is trained to do so, i.e. the distribution learned and reinforced is one such that it maps the I distribution to the O distribution that is what human beings typically do or want when writing unit tests. The design is the suite of algorithms used for this (which is ultimately global next token prediction)
I think you are conflating design (the mechanics) with their cultural or user level function.
You are basically saying "my bike isn't just rotating wheels on a frame because it takes me places". Hopefully you can see how both parts of that statement are true and why contrasting them as if the first clause is somehow false or inaccurate is incorrect.
Both these are true statements. They operate at different levels. People are constantly conflating these levels in the "is it a next token predictor" (the answer is yes) debate.
A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.