News for hackers. Stuff that matters.
Re:Data Compression Drives the Internet. Here’s How It Works by twotwotwo on Friday June 09, 2023 @07:18PM (#36265648)
There was a series of articles on lossless and lossy and compression techniques in, I think, PC Magazine that I read as a kid and made a big impression on me. I didn't, like, end up ready to write code to build Huffman trees or anything, but it did change it from a total black box into a bunch of smaller pieces each of which a mere mortal can understand.

The compression rabbit hole can be a rewarding to go down if you haven't. Besides the tech itself making a lot of things work better (like a Web built on human-readable formats, or more specialized stuff like Parquet or other columnar formats, or video on the lossy side), it can give you some tools or perspective that apply elsewhere, e.g. to other probablistic stuff like caching and predictions for lossless compression, or other signal-processing stuff for lossy compression.

Re:Data Compression Drives the Internet. Here’s How It Works by retrac on Saturday June 10, 2023 @09:30AM (#36270301)
> it can give you some tools or perspective that apply elsewhere

Compression is right at the junction of computer science and information theory. It might be what joins them together. For example, if you have infinite space, any algorithm can be O(n), trivially fast, with a giant lookup table between inputs and outputs. In practice you want to find a smaller but reasonably fast equivalent program to represent the same mapping -- optimizing a program is lossless compression. In machine learning, model training takes a large one-dimensional array (such as a string with all the text on the Internet) and lifts it into a multidimensional space. Language models create output strings patterned like the input string, while storing less data than the complete input string -- machine learning is a type of lossy compression.

Re:Data Compression Drives the Internet. Here’s How It Works by Folcon on Saturday June 10, 2023 @09:51AM (#36270462)
I found their Encyclopedia page[0], but I suspect you meant something better than this?

Any further recollections to help drill down and find it?

- [0]: https://www.pcmag.com/encyclopedia/term/data-compression

Re:Data Compression Drives the Internet. Here’s How It Works by twotwotwo on Saturday June 10, 2023 @08:16PM (#36276715)
It looks like it was Barry Simon in "Lab Notes" in PC Magazine of June and July 1993:

https://books.google.com/books?id=gCfzPMoPJWgC&lpg=PP1&pg=PA...

https://books.google.com/books?id=eX8w8B-OhIIC&lpg=PA371&pg=...

Re:Data Compression Drives the Internet. Here’s How It Works by MauranKilom on Saturday June 10, 2023 @10:01AM (#36270526)
Another interesting facet is that, according to some schools of thought, compression and AI are equivalent problems: The better you understand something, the less you need to memorize to reproduce it.

https://en.wikipedia.org/wiki/Hutter_Prize (and https://en.wikipedia.org/wiki/AIXI)

Of course, large language models are (by definition) currently going the other direction, but it remains to be seen whether that leads to artificial intelligence (whatever that ends up meaning).

Re:Data Compression Drives the Internet. Here’s How It Works by brookst on Saturday June 10, 2023 @11:55AM (#36271619)
Maybe a naive question: are LLMs really going the other way? My intuition is that the model weights are much smaller than the information encoded in them.
Re:Data Compression Drives the Internet. Here’s How It Works by floobertoober on Saturday June 10, 2023 @12:06PM (#36271701)
> Of course, large language models are (by definition) currently going the other direction ...

How so? Aren't the networks' weights orders of magnitude smaller than the training data?

Re:Data Compression Drives the Internet. Here’s How It Works by karpierz on Saturday June 10, 2023 @12:35PM (#36271985)
I believe the parent poster's point is: LLMs are more effective when they use more memory, meaning the less they are forced to compress the training data, the better they perform.
Re:Data Compression Drives the Internet. Here’s How It Works by govg on Saturday June 10, 2023 @12:33PM (#36271972)
I interpreted that statement as saying the current practice is to make LLMs larger and larger (so they effectively memorize more and more data) to make them more powerful, but from the perspective of information theory, if models were powerful and "understanding", then models could stay the same size and become more and more powerful as they get increasingly better at compressing the available information. I am not sure if this interpretation was what was meant though.
Re:Data Compression Drives the Internet. Here’s How It Works by nico on Saturday June 10, 2023 @12:41PM (#36272075)
> theory, if models were powerful and "understanding", then models could stay the same size and become more and more powerful

Check out Karpathy’s video “let’s build GPT from scratch”

Using the exact same data (a 1MB text file containing all of Shakespeare’s works), he goes from several models that spit out garbage, to a GPT model that outputs good looking garbage

The only changes are the algorithms used, the data is always the same

There is a big opportunity in creating small models that can “learn” and then “reason about”, very little data

Re:Data Compression Drives the Internet. Here’s How It Works by MauranKilom on Wednesday June 14, 2023 @03:16PM (#36331103)
But they don't losslessly recreate the training data.
Re:Data Compression Drives the Internet. Here’s How It Works by ta93754829 on Sunday June 11, 2023 @06:39AM (#36280056)
Once ChatGPT was trained on "the internet of 2021", how big is the remaining model? Is ChatGPT effectively a compressed version of the internet?
Re:Data Compression Drives the Internet. Here’s How It Works by ot on Saturday June 10, 2023 @12:05PM (#36271694)
Was it maybe Dr Dobb's? Mark Nelson had an excellent series of articles about compression which opened up that world for me. I ended up working on compression for my PhD many years later.

I can't find any archives for Dr Dobb's, but some articles are in his personal site, for example this one about arithmetic coding: https://marknelson.us/posts/1991/02/01/arithmetic-coding-sta...

Re:Data Compression Drives the Internet. Here’s How It Works by twotwotwo on Saturday June 10, 2023 @08:24PM (#36276760)
It looks like the ones I were PC Magazine, https://books.google.com/books?id=gCfzPMoPJWgC&lpg=PP1&pg=PA... and https://books.google.com/books?id=eX8w8B-OhIIC&lpg=PA371&pg=... .

Much appreciation to all the folks who put in the work to make these things interesting and accessible.

Re:Data Compression Drives the Internet. Here’s How It Works by oikawa_tooru on Monday June 12, 2023 @06:03PM (#36301886)
Hi I am a neophyte interested in compression. It's difficult to find communities regarding compression online. I am looking for a guide. Is there any place that I can dm you?