Re:Language Models Are Super Mario: Absorbing Abilities from...
by geon on Sunday April 07, 2024 @04:32AM
(#39959281)
Do you mean dictionary as just a list of words in alphabetical order, or with detailed definitions for each word?
Re:Language Models Are Super Mario: Absorbing Abilities from...
by nico on Sunday April 07, 2024 @10:37AM
(#39961080)
Detailed definitions for each word, like a physical printed dictionary book
Re:Language Models Are Super Mario: Absorbing Abilities from...
by geon on Sunday April 07, 2024 @04:34AM
(#39959293)
This video? https://youtu.be/kCc8FmEb1nY?si=QR-ADr0_Y6_P2QxC
What really stood out to me is that, while the typical wisdom is “more data -> better model”, in the video they use the exact same data for all the models, so all the gains are achieved only through improved algorithms and computation power
With that in mind, I wonder if at some point someone will figure out an algorithm for a model that can be trained on just an English dictionary and get a decent LLM-equivalent model that can have a basic conversation. Given the small size of the data (a dictionary), I assume the model would be pretty small and quite fast to run, even on older or smaller machines