News for hackers. Stuff that matters.
Re:Data Compression Drives the Internet. Here’s How It Works by floobertoober on Saturday June 10, 2023 @12:06PM (#36271701)
> Of course, large language models are (by definition) currently going the other direction ...

How so? Aren't the networks' weights orders of magnitude smaller than the training data?

Re:Data Compression Drives the Internet. Here’s How It Works by karpierz on Saturday June 10, 2023 @12:35PM (#36271985)
I believe the parent poster's point is: LLMs are more effective when they use more memory, meaning the less they are forced to compress the training data, the better they perform.
Re:Data Compression Drives the Internet. Here’s How It Works by govg on Saturday June 10, 2023 @12:33PM (#36271972)
I interpreted that statement as saying the current practice is to make LLMs larger and larger (so they effectively memorize more and more data) to make them more powerful, but from the perspective of information theory, if models were powerful and "understanding", then models could stay the same size and become more and more powerful as they get increasingly better at compressing the available information. I am not sure if this interpretation was what was meant though.
Re:Data Compression Drives the Internet. Here’s How It Works by nico on Saturday June 10, 2023 @12:41PM (#36272075)
> theory, if models were powerful and "understanding", then models could stay the same size and become more and more powerful

Check out Karpathy’s video “let’s build GPT from scratch”

Using the exact same data (a 1MB text file containing all of Shakespeare’s works), he goes from several models that spit out garbage, to a GPT model that outputs good looking garbage

The only changes are the algorithms used, the data is always the same

There is a big opportunity in creating small models that can “learn” and then “reason about”, very little data

Re:Data Compression Drives the Internet. Here’s How It Works by MauranKilom on Wednesday June 14, 2023 @03:16PM (#36331103)
But they don't losslessly recreate the training data.