Re:Tiny Language Models Come of Age
by whimsicalism on Sunday October 08, 2023 @12:57PM
(#37812227)
60% is not vastly smaller in the way that I meant - we are talking about differences of three orders of magnitude in this article and that is what I am skeptical of.
Nevertheless, I have not actually seen a model exhibiting high-level LM capabilities (of the GPT-4 level) that was distilled.
Just because a model can match in perplexity with fewer parameters is not the same as matching the performance as subjectively measured by humanity interacting with the model.
Re:Tiny Language Models Come of Age
by klyrs on Sunday October 08, 2023 @05:30PM
(#37814590)
Early days, yet. If we could cut the size by 60% every few years, we'd be looking at a situation akin to Moore's law where a performant AI might fit in my grandkid's pocket.
Nevertheless, I have not actually seen a model exhibiting high-level LM capabilities (of the GPT-4 level) that was distilled.
Just because a model can match in perplexity with fewer parameters is not the same as matching the performance as subjectively measured by humanity interacting with the model.