Techniques like distilling show that you can in fact get similar performance with 60% fewer parameters. Yes it depends on making the large model first, but that only implies that somehow the large model is perhaps more useful in initial learning. And if that's the case, then it would be rather surprising if we couldn't identify why and create new training techniques that go directly to the smaller equivalent model.
So in this way you've already lost your bet, as we already know that it's possible to create vastly smaller models with similar performance.
Re:Tiny Language Models Come of Age
by whimsicalism on Sunday October 08, 2023 @12:57PM
(#37812227)
60% is not vastly smaller in the way that I meant - we are talking about differences of three orders of magnitude in this article and that is what I am skeptical of.
Nevertheless, I have not actually seen a model exhibiting high-level LM capabilities (of the GPT-4 level) that was distilled.
Just because a model can match in perplexity with fewer parameters is not the same as matching the performance as subjectively measured by humanity interacting with the model.
Early days, yet. If we could cut the size by 60% every few years, we'd be looking at a situation akin to Moore's law where a performant AI might fit in my grandkid's pocket.
Re:Tiny Language Models Come of Age
by fnordpiglet on Sunday October 08, 2023 @12:59PM
(#37812243)
That doesn’t mean that a much larger with high quality training data and new training techniques won’t be more powerful than a smaller model. We have incredible efficiencies in modern computers in addition to immense resources. They both are needed for the best performance. I suspect it’s the same for LLM. Superior data, superior techniques, and superior scale will yield superior models.
Information density will increase in models just as spatial density increased for hardware.
Re:Tiny Language Models Come of Age
by whimsicalism on Sunday October 08, 2023 @04:49PM
(#37814275)
Saying that information density will increase is very different from saying model sizes will decrease of a similar order of magnitude.
Regardless, this isn't the trend we've seen with software RAM usage - information density is only decreasing as hardware becomes more abundant and comparatively less engineering effort goes into fitting everything into as small a portion as possible.
Modern computers don’t have less ram than the old fashioned room size computers. Amount of RAM seems much more analogous here than the size of the hardware running the software. Of course the latter will go down, but the former? I don’t see why.