War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens long.
Many people could reason about the book in its entirety. They would not have an encyclopedic recall of random dates and side characters or be able to quote any passage, but they could perform deep analysis on it.
I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps with some loss of fidelity.
My guess is that there will need to be an advancement or two before we can can get an AI to read all of the ASIOF books so far and ask "What really happened at the Tower of Joy?"
> At the Tower of Joy, Ned Stark defeated three members of the Kingsguard and discovered his dying sister, Lyanna, who made him promise to protect her son, Jon Snow, whose true parentage remained a closely guarded secret.
Seems like ChatGPT-3 already knows, unless there's a deeper secret that I'm not deep enough into ASIOF fandom to know about.
There's a bunch of prior art for adaption techniques for getting new data into a trained model (fine tuning, RLHF etc). There's no real reason to think there won't be more techniques that turn what think of now as the context window into something that alters the weights in the model and is serialized back to disk.
But also, not just ASIOF is in the training set, but presumably lots of discussion about it and all the interesting events in the book.
Some of them, some of the time. This is best comparable with ChatGPT having those books in its training dataset.
The context window is more like short-term memory. GPT-4 can fit[0] ~1.5 chapters of Game of Thrones; GPT-4-32k almost six. Making space for prompt, questions and replies, say one chapter for GPT-4, and five chapters for GPT-4-32k.
Can you imagine having a whole chapter in your working memory at once? Being simultaneously aware of every word, every space, every comma, every turn of phrase, every character and every plot line mentioned in it - and then being able to take it all into account when answering questions? Humans can do it for a paragraph, a stanza, maybe half a page. Not a whole chapter in a novel. Definitely not five. Not simultaneously at every level.
I feel in this sense, LLMs already surpassed our low-level capacity - though the comparison is a bit flawed, since our short-term memory also keeps tracks of sights, sounds, smells, time, etc. and emotions. My point here isn't really to compare who has more space for short-term recall - it's to point out that answering questions about immediately read text is another narrow, focused task which machines can now do better than us.
----
[0] - 298000 words in the book (via [1]), over 72 chapters (via [2]), gives us 4139 words per chapter. Multiplying by 4/3, we get 5519 tokens per chapter. GPT-4-8k can fit 1.45x that; GPT-4-32k can fit 5.8x that.
[1] - https://blog.fostergrant.co.uk/2017/08/03/word-counts-popula...
[2] - https://awoiaf.westeros.org/index.php/Chapters_Table_of_cont...
We take out key points explicitly so it's not summarized, and for the rest (less important parts) we summarize it and save it.
That would very likely fit and it would probably yield equal to or better recall and understanding than humans.
That in essence, I believe, is the difference. An LLM (while still predicting the next word, given it's context), seem to care less about the number of subjects in a given context, than humans do.
[1]: https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus...
Summarizing chapters with the very same LLM and including that as context may very well get you far.
That’s analogous to having War and Peace in the training set. When actually reading War and Peace, nobody‘s working memory includes a precise recall of everything they’ve read thus far, which is more analogous to an LLM’s context size.
> the linear attention of RWKV leads to significant efficiency gains but still, it may also limit the model’s performance on tasks that require recalling minutiae information over very long contexts. This is due to the funneling of information through a single vector representation over many time steps, compared with the full information maintained by the quadratic attention of standard Transformers. In other words, the model’s recurrent architecture inherently limits its ability to “look back” at previous tokens, as opposed to traditional self-attention mechanisms. While learned time decay helps prevent the loss of information, it is mechanistically limited compared to full self- attention.
> Another limitation of this work is the increased importance of prompt engineering in comparison to standard Transformer models. The linear attention mechanism used in RWKV limits the information from the prompt that will be carried over to the model’s continuation. As a result, carefully designed prompts may be even more crucial for the model to perform well on tasks
Sadly too many folks copy and paste what works for openAI and move on when it fails
> we loaded the entire text of The Great Gatsby into Claude-Instant (72K tokens) and modified one line to say Mr. Carraway was “a software engineer that works on machine learning tooling at Anthropic.” When we asked the model to spot what was different, it responded with the correct answer in 22 seconds.
Hopefully this gets the point across without the mathematics and a precise description.
Vs
I need to memorise everything as it’s spoken out, and then answer on it
Is a good approximate on the difference.
The kicker though as people pointed out, there are individuals on earth who can memorise things to great length. So the same will apply here (need more training)
It’s also why it is recommended by the community to ask the question first, then put your context document. In a document QnA task
For example, if someone says, "I should be a better mother", and hold this impression in their mind over time, the associations their brain makes with that image and various behaviors can be used as a guiding force during the physical reorganization which occurs when one creates a new habit, or adjusts a behavior or muscle memory.
And these associations depend upon observations made by the individual, which they have either been taught or innately know can be categorized as mother-like. The brain is typically trained to make this association in the case of the mother by mirroring and attaching to the first female who exhibits nursing behavior. These archetypes are a visual representation of deeper mental models which we are primed to carry on between generations.
And then I wondered how this might apply to neural models, if some system of archetypes naturally emerges, if they're useful for programming or memorization/compression, or how we might explore imbuing existing models with an archetypical system.
And this apply to all models. So when a specific model now does badly even at 32k or 50k it’s hard to say if it’s an architecture design issue, or a dataset issue
These things are very different. The mental model I have is that they put the entire book into a big hypercube and then *flash*, instantly, out comes the next token. And likewise, when they encounter text that is 8001 tokens long, token #1 is completely erased, gone, like it never existed.
You're confusing your perception of how your mind works with how your mind actually works.
Simply put, we have no idea how the human mind works. For all we know, its underlying principles could be very similar to LLMs, or they could be something nobody has thought of yet. But under no circumstances is human intuition about the human mind a reliable indicator for what is actually going on.
This is like asking ChatGPT why it came up with a specific response. It will give you an answer, of course, but that answer is generated the same way all its answers are – ChatGPT doesn't suddenly turn on a magic "introspection mode" that allows it to examine its internals. And there's no reason to assume that humans have such an introspection mode either. In fact, many results from experimental psychology seem to indicate that humans don't understand their own mental operations at all.
What we cannot do is make definitive claims about neurological processes and structures based on what we know about our cognitive processes.
Is there any actual evidence that our so-called reflections on our own thinking are anything more than hallucinations?
"It's obvious" doesn't count as evidence.
These studies confirm our perception that we can forget stuff.
One is a general statement of fact, the other implies introspection of one's own individual cognitive processes. I have yet to see evidence that the latter is actually possible.
The phenomenon of people forgetting stuff is not subjective. It's not one person in isolation thinking they just forgot something. Forgetting is a phenomenon on which there is a lot of social feedback and repeatable experiments.
You seem to be denying the possibility of ever connecting science back to individual perception. Denying this makes any and all science completely meaningless though.
It's not limited to observations about ourselves. You could ask the question "but aren't you hallucinating?" about absolutely everything required to verify the outcome of a physics experiment for instance.
LLMs notwithstanding, I have yet to see a replication of the human mind created by humans. Therefore, I would take any claim that neuroscience 'understands' how the human mind works (a claim that I've never heard from an actual neuroscientist) with a huge amount of salt.
Not that it matters, but for various reasons I have spent quite a lot of time around neuroscientists and I think most of them would agree that "no idea" is a pretty ridiculous way to characterize the state of the field.
We won't know before we've tried it. Reasoning by analogy with humans is not useful. In a year we'll have tried lots of things and can give a much better answer.
My theory is that the model will start generalising the information stored once it starts going past its limits (like real life humans)
So if we take books as an example, we probably do not remember every single word. But we might vaguely remember which section or book events occur
Combine this with the agent model, and we may have an alternative for embeddings. Where we can ask which pages the model recall is relevant to the question. Bring those pages up again. And get the answers
(Once again like how a human might answer in real life)
What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding.
You can view it as every token embedding mixing information from other embeddings into itself. Done ~100 times in parallel ("attention heads"), ~100 times in sequence (layers). As per GPT-3 model (175B).
But since this was derived from apple lite attention paper. The name is gonna stick, due to a lack of better alternative
Every-time someone comes up and say "this is not attention, because it does X and not Y"
My response is, ok, what would you call it then? Because no one (including me) seem to to able to find a better term, that fits its use case.
Hallucination used to mean the following. A basic neural network is:
f(x) = y = repeat(nonlinearity(ax[0] + bx[1] + ...))
And then you adjust a, b, c, ... until y is reasonable, according to the cost function. But look! The very same backpropagation can adjust x[0], x[1] ... with the same cost function and only a small change in the code.
This allows you to reverse the question neural networks answer. Which can be an incredibly powerful way to answer questions.
And that used to be called hallucination in Neural networks. Instead of "change these network weights to transform x into y, keeping x constant" you ask "change x to transform x into y, keeping the network weights constant".
Now it's impossible finding half the papers on the this topic. AARGH!
It's most famous as a problem with ChatGPT specifically, which is presented as a chat interface where an agent answers your question. In that context, it makes sense to think of confident and detailed wrong answers as hallucinations.
You could say that it's still an LLM underneath, but then you'd be talking about a layer of abstraction beneath what most people interact with. Given that they tap into the mental model of "chat with an agent" heavily with their interface and interactions, having small disclaimers and saying "I'm just an LLM" from time to time aren't sufficient to counter people's intuitions and expectations.
> it takes a fraction of every other token embedding and adds it to itself.
> every token embedding mixing information from other embeddings into itself
Noting the use of the word every. Phrased this way, calling it "attention" hardly makes sense, as attention is typically focused on something specific at any given time - not always on the entire thing.
The mixing step, is computed with the trained weights, meaning the model does learn on its own, when/what to emphasise in this "mixing" process.
Hypothetically speaking, if a token does not matter, it mixing weights could end up being literally 0 (or something close to it)
This could be seen as "soft attention" where in theory there would be few winners for every "attention head".
It's also possible that the only purpose the softmax actually serves (or at least major component of) is that of normalization. Without it, the variance in the internal layer dynamics between training samples would be fairly large (some training samples may have tokens that interact heavily, while others not). Making optimization problematic.
And, in the case of BERT, every token embedding after it too.
Details aside, math suggests they _must_ be using some sparse attention method: the memory used by attention is O(l^2 * d) (here O notation is a bit of overkill: it's exactly that number in 8 bit quantization or twice that in 16 bit floats). With l=32k, and d probably in the order of a few k (even "old" BERT model had it at 768, it only went up since), that would be of the order of a few TB. For (a piece of) _a single_ layer, they probably have dozens, if not hundreds, of it. The largest GPUs in commerce have 80GB of memory; there's no way they are really using that many GPUs for each single layer (even leaving aside the fact that runtime would probably be absolutely horrible).
Implicitly or explicitly, the transformer must learn to have both a "summarized" version of attention (that tells it where to focus on) and more "detailed" ones. It doesn't need to be specifically coded, if using sparse attention, the model might for example get different levels of attentions at different layers, somehow, but they probably have coded something a bit smarter than that.
[1] https://kir-gadjello.github.io/posts/gpt4-some-technical-hyp... [2] https://arxiv.org/abs/2111.12763
Machine learning is optimizing mathematical weights and biases, and the human mind is messy wetware. One of the first things you will read in any machine learning textbook is to throw away the notion of AI trying to simulate the human mind.
It’s not clear how to learn this. Transformers work so well because there are only a few layers between input tokens and output. Adding a summarization step is a whole new research project.
Note that what people claim here is not true: just because you go past the context window does NOT mean everything is forgotten, it means that you're back to RNN levels of performance.
In transformers, just like in RNNs, processing the nth token influences the nth token AND the (n+1)th token. So the influence of the nth token "slowly dies down". It influences the (n+1)th token a lot, the (n+2)th token a bit less and so on. The special thing about attention is that it doesn't start dying until you go past the context size.
Or with a bit of math notation. If a neural network is a function f, then:
RNN: f(x[n]) = f( f(x[n-1]), x[n] )
Transformer: f(x[n]) = f(sum(i=0...context size, f(x-i)), x[n])
The difference between LSTMs and GRUs is in the structure of f. And "sum" is, like a lot of things in here, uh, basically accurate but missing a lot of detail.
In your example, x[n-1] = f(x[n-2], x[n-1]). So really, your expression should be
RNN: x[n] = f( … f( f( f( f(k0, x[0]), x[1]), x[2]), x[3]), …, x[n-1])
where k0 is a learnable parameter.
All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer questions about the text coherently.
But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases. They must start to generalize. A human could not read Infinite Jest in one pass and then answer details about every single sentence. But could a transformer, or a RNN? As the context grows, is it harder to keep a high granularity of detail? Or am I wrong in trying to think of these models the way I think about the human mind, and they are actually able to handle this problem just fine?
I'm aware that we can cheat a bit, by adding a lookup step into an embedding database, to provide "infinite" context with "infinite" precision. But to me, that is analogous to a human looking up information in a library in order to answer a question. I'm interested in the inherent, emergent memory that these models have available to them in just one forward pass.