That said, there are many works attempting to increase the compute utilization of transformer language models (early exit, mixture of depths) and novel architectures (SSMs etc.).
I think I’ve read something about this but I wonder if you could abstract attention to sentence/page levels and then only recalculate the parts that are relevant.
I.E. the KV cache is 'just' a time saving measure because an LLM goes back and calculates those values anyway. (Which is why per-token compute increases exponentially otherwise)
You're not wrong that you could make an LLM more stateful. There are plenty of ideas for that but it would
a) be far more compute intensive to train and run (especially train)
B)be susceptible to all of the issues that RNNs have.
C) most importantly, it would almost certainly just converge at scale with transformers. Labs run small scale, internal tests of architectures all the time and most of them basically come to this conclusion and abandon it
By looking at that tree some basic filtering is done. Such as picking the branch that has the highest summed confidence, or the branch that has the fewest repeated tokens, or the fewest tokens that match with input tokens, or more often some combination of the above, plus a random choice weighed by summed confidences.
That how you can give a LLM with complete fixed weights, which is all LLM, the same input multiple times, but get different outputs.
So to answer your specific question, it can “change its mind”. Every token produced creates a new opportunity for the stochastic output filters to pick a new path through all the possible outputs.
Which is my understanding of how they work and the dynamic at play.
I'm a total amateur in this field though.
If you want to just see the tokens for yourself, though, just enter some text here:
my gut said "pig" and "ment" on this one, which happens to be right, but my gut would also say "para" and "graph" but no, "paragraph" is a single token which falls way outside the "normal" length I see of 3-4 characters
In either case, I do consistently see spaces between characters included as part of the token following the space.
" paragraph" (10 characters) is the longest token I've seen- and now I wonder what the longest token is
Bananas are yellow due to a
Bananas are yellow due to an
In the first case it might respond Bananas are yellow due to a pigment called bromelain.
In the second case it might respond Bananas are yellow due to an organic compound called bromelain, which is a yellow pigment.
So in either case GPT could have picked "a" or "an" without any impact on the semantic meaning of its response. In the extreme case, you could see the LLM operating according to a dumb heuristic: The token following "due to" is "a" with 55% probability, "an" with 45% probability.
In reality it is of course more sophisticated than this. But this dumb heuristic would explain the behavior.And if you didn't actually include any facts about bromelain in the pretraining data, LLMs absolutely could autocomplete this with something about "an optical illusion." GPT-3 made factual mistakes like that pretty routinely, but I recall it figured out the grammatical rules of "a" and "an."
I don't think the concept actually needs to be pre-activated as you said, though I agree with faabian that this "preactivation" probably does happen in some implicit/emergent sense.
2. Bananas are yellow due to a specific carotenoid called beta-cryptoxanthin, which gives the fruit its characteristic yellow hue.
3. Bananas are yellow due to a gradual increase in the concentration of carotenoid pigments as the fruit ripens and chlorophyll levels decrease.
4. Bananas are yellow due to a series of enzymatic reactions that convert starch into sugars and break down the green chloroplasts, revealing the underlying yellow carotenoids.
5. Bananas are yellow due to a change in the pH levels within the fruit cells during ripening, which triggers the production of yellow carotenoid pigments.
6. Bananas are yellow due to a genetic trait inherited from their wild ancestors, which enabled the development of carotenoid pigments as a way to attract seed dispersers.
7. Bananas are yellow due to a complex interplay between various plant hormones, such as ethylene and abscisic acid, which regulate the ripening process and pigment formation.
8. Bananas are yellow due to a metabolic shift from chlorophyll synthesis to carotenoid synthesis as the fruit reaches maturity.
9. Bananas are yellow due to a natural defense mechanism that involves the production of carotenoid pigments, which protect the fruit from oxidative stress during ripening.
10. Bananas are yellow due to a evolutionary adaptation that helps the fruit stand out against the green foliage, making it more visible to potential seed dispersers.
The output of an LLM is usually randomly sampled from the top few highest probability next token/word predictions, but the model itself has no idea which word the sampler will pick. It presumably has some conceptual plan of what could follow "a", or any of it's other suggestions, but any such plan (high level prediction) is then rethought from scratch once "a" is generated.
The model not only can, but has to, change it's mind after each word generated, so this "planning ahead" is very ephemeral - more like a freestyle rapper making it up on the fly than someone thinking deeply about how best to reply and how to express it.
Models do struggle with these tests, for sure, but from an analytical standpoint, a "next token predictor" should not be able to ever correctly land on the right token 100 tokens in the future.
Edit: Thinking about it, I suppose it is possible that the model can encode a "destination" in the first token. Like a pool shot that is artfully bounced off many bumpers to hit a ball, perhaps the LLM can encode a "path" to a destination token in the first token generated. Which might be even crazier as it suggests that the model is playing a meta-game with being able to precisely manipulate the individual layers of output, even though those layers are disparate from token to token.
One way to think of what all the intermediate layers are doing is to consider them as levels of a linguistic parse tree with the leaves (words) at the bottom and trunk ("sentence") at the top, except in the transformer the evolving embeddings at each level contain semantic as well as syntactic information. This largely hierarchical view of language was the motivation for the transformer design.
It seems we should really think of each layer of the transformer as an independent predictor, with increasingly abstract and more semantically complete information available as we ascend the transformer layers towards the output. Predict next token is only what the transformer is being trained to do at the output layer. At the inner layers (i.e. the bulk of what the transformer is doing), it will be predicting at these higher levels of representation held at those layers.
These models do struggle (although getting better) at ending on a given word rather than starting on it, and understandably so since random sampling and continual resetting after each output token (= new input sequence) means that planning ahead at level of word specificity is simply not an option. They have to continually adapt to next sampled token, and take it from there.
I'm guessing that ability to end on a chosen word is due to continued salience of that word during generation, prediction of sentence fragments using/ending with that word, and opportunistic stopping when it has been emitted and the sentence is complete. Kind of the same way you might do it yourself if you just started talking immediately without planning, while trying to end on a given word.
Did an LLM really make this basic grammatical error?
That's what makes it surprising.
Lets say you ask an LLM
And it replies I would think that the concept of "pigment" and "bromelain" are already somehow activated in the neural net when it outputs "a". Because now it can't change its mind anymore and follow up with "an optical illusion that makes humans perceive every bent object as yellow". So it seems to have already planned ahead to talk about the pigment called bromelain.Would it be possible to capitalize on the work that has already been done when the LLM outputs "a"? Could the state of the neural net be somehow preserved for the next answer?