Re:Data Compression Drives the Internet. Here’s How It Works
by nico on Saturday June 10, 2023 @12:41PM
(#36272075)
> theory, if models were powerful and "understanding", then models could stay the same size and become more and more powerful
Check out Karpathy’s video “let’s build GPT from scratch”
Using the exact same data (a 1MB text file containing all of Shakespeare’s works), he goes from several models that spit out garbage, to a GPT model that outputs good looking garbage
The only changes are the algorithms used, the data is always the same
There is a big opportunity in creating small models that can “learn” and then “reason about”, very little data
Check out Karpathy’s video “let’s build GPT from scratch”
Using the exact same data (a 1MB text file containing all of Shakespeare’s works), he goes from several models that spit out garbage, to a GPT model that outputs good looking garbage
The only changes are the algorithms used, the data is always the same
There is a big opportunity in creating small models that can “learn” and then “reason about”, very little data