edmara
It's called the bitter lesson for a reason. Nobody likes seeing their life's work on some unicorn architecture get demolished by simplicity+scale
1. Tight integration necessary for fundamental compute constraints like memory latency.
2. Economies of scale
3. Opportunity cost to AI orgs. Meta, OpenAI etc want 50k h100s to arrive in shipping container and plug in so they can focus on their value-add.
Everyone will have to readjust to this paradigm. Even if next get AI runs better on CPU, Intel won't suddenly be signing contracts to sell 1,000,000 xeons and 1,000,000 motherboards etc
Also, Nvidia have 25bn cash in hand and almost 10 billion yearly r&d spend. They've been an AI-first company for over a decade now, they're more prepared to pivot than anyone else
Edit: nearly forgot - Nvidia can issue 5% new stocks and raise 100B like it's nothing.
I.E. the KV cache is 'just' a time saving measure because an LLM goes back and calculates those values anyway. (Which is why per-token compute increases exponentially otherwise)
You're not wrong that you could make an LLM more stateful. There are plenty of ideas for that but it would
a) be far more compute intensive to train and run (especially train)
B)be susceptible to all of the issues that RNNs have.
C) most importantly, it would almost certainly just converge at scale with transformers. Labs run small scale, internal tests of architectures all the time and most of them basically come to this conclusion and abandon it
>It replays the data to us and we suppose the LLM must have the property that generates this data originally.
So to clarify, what you're saying is that under the hood, an LLM is essentially just performing a search for similar strings in its training data and regurgitating the most commonly found one?
Because that is demonstrably not what's happening. If this were 2019 and we were talking about GPT-2 it would be more understandable but SoTA LLMs can in-context learn and translate entire languages which aren't in their dataset.
Also RE inference time, when you give transformers more compute for an individual token, they perform better https://openreview.net/forum?id=ph04CRkPdC
If the KQV doesn't encode information about likely future token sequences then a transformer empirically couldn't outperform Markov text generators.