A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.
Which is interesting, given that life itself is merely that. It's just that humans decided that that ain't good enough and invented something alien to nature - logic.
Hm. I would actually go so far as to say "predict" is just the wrong word to use in 2026, as it drags in a picture of what LLMs are that's from the pre RLVR era, and which is frequently used to suggest they have all sorts of limitations that they've now surpassed ("an LLM will never solve a major open mathematics problem, they're just predicting tokens"). These days I tend to use the more neutral phrase "generate the next token"; I'd only say "predict" in the technical context of (say) the pre-training part of the training run where the object is indeed to minimize the cross-entropy loss function by predicting next tokens in existing text. Generating a token with a modern LLM is better described as something like, "given the totality of its training, including not just next token prediction with pre-training but also solving vast numbers of math and coding problems with RLVR, the LLM generates tokens one by one."
Anyhow. I'm not sure the weight understanding is all that strange. (It's impressive, but so are many thing LLMs do these days.) LLMs are very very good at coding and math problems, of which this is one, and in particular, their RLVR training presumably includes many tasks involving neural networks. It's amazing and impressive, I agree, but I'd say less amazing than proving major open theorems in mathematics, which we already know they can do.
Substitute ‘generate’ and the meaning is unchanged.
It's worth reflecting on the original domain jargon for selecting the next token: completion.
And yes, I understand "prediction" is a standard term in the field. I just think it is a bad one and we should stop using it.
This doesn't make it any less fascinating that by combining this basic "do as I say" training with massive amounts of all of humanity's data, you can solve major problems—that is still incredible. But that doesn't fundamentally change what is going on, mechanistically.
I don't understand these pseudo-religious desires to read into models something "more" than what we know. We built them. We don't go around making wild claims about how cars are actually more than just a mechanical system, how they aren't just taking us places—we do, but usually such points are cultural not about the mechanica capabilities of cars. People in the LLM space seem to be constantly conflating these two views. It's entirely accurate to say, on the one hand, next token predictors combined with RL, and massive troves of existing human work, can solve culturally interesting problems, and to hold, on the other hand, that mechanistically what is happening is statistical next token predictions over global linguistic space and more or less "monkey see monkey do".
Mathematics models the world, it does not have substance. It is a useful way of looking and a useful way of manipulating, but many models are equally useful. Capital T Truth, beyond models is very difficult to get at. We can show that things are true for certain models well enough. Showing that something is true in all models (a universal truth) is far more difficult. You face incommensurability.
I see a lot of people in technology fall into this extremely basic logical fallacy called Reification constantly.
I mean, when you really get down to it, either the particles and such operate on very specific rules, and the rules govern how all our brain and body work. Followed to the conclusion, the universe is computable and we can simulate life.
You can demonstrate definitively that there nothing other than "next token prediction" by running the same inference engine/model/prompt/settings multiple times in a row on the same hardware at temp 0 with a seed set.
If you turn the temp up, then you might let the "mechanism" select pseudorandomly from a probability distribution of k-likely tokens with minimum p-cumulative probability. Even then, if the next token has a very high probability, deviation will be uncommon.
I just performed this experiment right now on a few dozen runs of a dense model at thousands of tokens of output at temp 0, and all were identical. Then I did it with a MOE model, which might get some benefit of picking different experts each time. Nope. Outputs were again identical.
Describing LLMs as next token predictors isn’t incorrect, it’s meaningless. It’s the equivalent of describing brains as “a bunch of atoms”.
The problem with these descriptions is that they say nothing about high-level capabilities. If you can predict the next token in a proof of the Riemann hypothesis, then you can prove the Riemann hypothesis.
The part that really bakes my noodle: `/foo/bar/baz.txt` can be a filename that no one has ever typed before, the error inside the file can be a typo that no one has ever typed before, and it will still correctly "predict the next token" to solve your problem.
In the case of the brain, it'd be more akin to the reduction of the brain to just the neuronal dynamics (obviously experience phenomenally is much more than that to us!)
But both views can be useful, depending on what you are trying to understand.
The problem with these arguments is that people pick and choose based on their persons preferences. People who are in love with LLMs will, on the LLM side, argue against reductionism "they are way more than next token predictors!" then they'll turn around and equally try to argue "the substrate doesn't matter! The brain is "just" neurons and digital activity so of course we can replicate it!" (this is also incorrect, btw, the brain is also an analog system, even in its electrical connections and not just in its chemistry)
The on the other side the anti LLM crew will do the opposite (launch into reductionism when it comes to LLMs, resist it when it comes to humans). I'm a human being and I like humans and I think people have adopted a strange view of LLMs, which are a tool and should be treated as such, so I tend to side with the humans. The stalwart defenders of LLM honor that would balk at any attempt to reduce them in stature to what they fundamentally are, mechanistically, are... an odd bunch. I'm confused why people are so passionate about it. If someone told me "well your IDE is just a fancy AST walker" yeah I think that is a dumb take but I wouldn't feel like I had to argue against such a take either saying "no! no! it's so much more!"
A system is not just its little bits.
Where "system" and "little bits" can be just about anything. But people endlessly repeat this mistake, and it has been a go-to misunderstanding/misrepresentation for a lot of people on this forum.
if it was designed to edit code, id expect the first translation from latent space out to tokens to be the final code, not be assembling the code by random number generator.
its not really designed to edit code until the tests succeed, we try to manipulate the distribution of tokens to predict by prompt engineering with the test results.
Going back to first principles, wikipedia mentions [0] inference as:
> "Inferences are steps in logical reasoning, moving from premises to logical consequences. Inference is traditionally divided into deduction and induction"
Is that exactly what next token predictor means?
Training is not design. Design is the mechanism, namely a statistical method of learning distributions, and of executing those distributions computationally as an IO function.
The model is not designed to write unit tests. It is trained to do so, i.e. the distribution learned and reinforced is one such that it maps the I distribution to the O distribution that is what human beings typically do or want when writing unit tests. The design is the suite of algorithms used for this (which is ultimately global next token prediction)
I think you are conflating design (the mechanics) with their cultural or user level function.
You are basically saying "my bike isn't just rotating wheels on a frame because it takes me places". Hopefully you can see how both parts of that statement are true and why contrasting them as if the first clause is somehow false or inaccurate is incorrect.
Both these are true statements. They operate at different levels. People are constantly conflating these levels in the "is it a next token predictor" (the answer is yes) debate.
To me this is fundamentally different from having ingested a bunch of docs so it now “knows Java” in its weights.
it probably has however been trained on papers talking about ways they extracted subsets of LLMs as say, the part that knows chess, and would be able to then use similar commands to process the weights
There are classes of adversarial attacks targeting them that don't work as well on us, and vice versa.
It's like a sorting algorithm, which I can implement in many different ways in just a few dozen lines. If you look at the binary code (so like raw weights) of the compiled program, you will see just a bunch of a different numbers, but if you read the code, in most cases you will understand that it's a sorting algorithm. So it seems they just operate on different levels of abstraction that are not available to our brains.
do they break down to a finite subset of groups?
In the case of models, the almost-theory (hypothesis with a good deal of evidence) is that once the amount of training data exceeds what the model can simply represent internally, the best way to keep improving is to start to "understand" the data. Understand in this case means to form internal control structures and decision systems that represent the data algorithmically rather than just memorizing it.
I put understand in quotes because we don't know how far this goes toward what we call understanding, or whether there's still a lot missing.
Its not natively reading the weights in the Matrix “symbols scrolling on a screen” sense. Still bloody impressive and not an outcome I would have bet on!
Yes, but everything, including the human brain, is "fundamentally" some underlying simpler primitives.
>E.g., “breakout game weights” always look the same regardless of training and model?
Wouldn't this be expected to be roughly the case, given NNs that share the same basic architecture to model the same quite simple thing like a breakout game?
The prompt told Astral what the input and output layers mean -- that is, how the current state of the game is turned into a vector for matrix multiplication, and how the result of those multiplications is turned into an action to take in-game.
Astral saw that the model was trained on a game where the player is at the bottom of the grid and moves left and right, reward objects live at the top, and there's some not-player-controlled thing moving around in the middle. Astral knows that this type of game is called "breakout."
A model that plays breakout will have meaningful weights for the player being in the bottom row, but not for the player being anywhere else, because the player can't go anywhere else. Astral saw that there were meaningful weights for the player in the bottom row, and nodes representing the player in any other row had low-magnitude noise, so the model had never been trained on the player being anywhere besides the bottom row.
Similarly, breakout has "reward" objects in the top rows, and "there is a reward object at [COORDINATE]" is part of the input layer. Astral saw that the model was trained on reward objects in the top few rows.
A couple of highlights to pique people's interests:
- Attention is kind of like "Global Illumination" for next tokens (they both have a game dev background)
- Attention is constantly being recalculated _for the next token_. This part never "clicked" before.
- LLMs are "next token predictors" in the same way that games are "next frame predictors".
That's exactly what Platonic Representation Hypothesis would predict.
There also have been plenty of studies showing that various neural networks, including LLMs, develop implicit world models that reflect the geometry/structure of a task they're trained on. A trained "chess engine LLM" would have a "chessboard-shaped" world model inside it.
So, if you can figure out what the geometry of a task is, you can search the weights for correlates of that geometry. Then you can try to improve those correlates so that they track the task better.
Seems like bleeding edge LLMs might already be superhuman at "figure out geometry from looking at weights".
Even if a slight rotation passing through a nonlinearity isn't good enough, you can just reorder the vector components which makes literally no difference.
A "reorder the vector components" transform preserves the similarity perfectly. So does a transform that maintains orthogonal equivalence.
These models are trained on pirated textbooks and papers, including every ML paper ever written.
I am amazed people still don't know the difference between the shape of data (token prediction) and the actual problem (learning whatever is required to be able to predict the next token).
The former does not limit the sophistication of the latter in any way.
(And you provided a valuable service/example for everyone who just thought the same thing! That phrase just keeps showing up. Whack! Whack! /h)
Unexpected free feature that plays right into recursive self-enhancement. What happens when they are trained to scale this ability up?
This could hit a wall, or be a step change in ... the rate of step changes.
To me “amazing” is not the right level of speechlessness. I mean, an LLM is still fundamentally a next token predictor, right? So what does this imply? That weights have some universal properties? E.g., “breakout game weights” always look the same regardless of training and model? Also, how does this meta-understanding of the weights end up in the weights in the first place? Idk, this is mind-blowing to me.