LLM's would make excellent (but very slow) lossy or lossless compressors.
> I read your page "http://www.cs.fit.edu/~mmahoney/compression/rationale.html" and found it very interesting. As an AI grad student, I never really thought of data compression as equivalent to AI.
>After reading your page, I have a problem with the wikipedia contest. The contest is trying to achieve something "intelligent," but penalizes large decompressors. If the decompressor can be thought of as the "brain" that we are trying to make intelligent, this is penalizing a large brain. The human brain, the pinnacle of intelligence as we know it, evolved to contain a huge amount of memory. The real-world never penalized better human brains during the brain's evolution. You have a larger brain? Great, you're smarter, and will probably fare better!
>That is my philosophical argument. My practical argument is using the size of the decompressor does not scale with the contest. If the contest was on 10 GB, the size of the decompressor would suddenly matter 10x less (assuming compression size is linear to the input size). Isn't that somewhat arbitrary? I entirely understand why the size of the decompressor cannot be ignored, but in its current state I think the limitation is drastically hurting the chances of anything moderately "intelligent." Adding the size of the decompressor into the rules of the contest encourages hacks instead of something more cognitively intelligent-like.
>Perhaps a more fair rule would be to not penalize decompressors up to a certain point. For example, anything less than 200 MB is free. I think this would be a more fair compromise, because IMO truly intelligent artifical agents, whenever they arise, will have to use large amount of memory.
>I would appreciate to hear what you think of this analysis.
The reply I got:
> Good compressors will use lots of memory, but this is not the same as requiring a large decompressor executable. The program will typically allocate much more memory than its size. There are practical limits on memory like 1-2 GB but that is another matter. However we must include the decompressor size or else it is easy to write a decompressor to keep a copy of the data and compress to 0. This is easy even with a 200 MB limit, even for enwik9 by keeping a copy internally compressed with ppmd or such.
The real problem with the Hutter prize is that the data and compute are too small. It seems obvious now that the regime where compression turns into AGI is far beyond the scale of the current contest (1 GB data, 50 CPU hours). If you simply bumped those limits up five or six orders of magnitude then the Hutter prize could be relevant today. But you'd also have to increase the size of the prize a few orders of magnitude because the cost of competing would be more than the current prize money.
Of course a prize isn't really a necessary incentive to develop AGI, since getting an AGI is about the best incentive you could possibly imagine.
Could you elaborate on why that is. It seems very non-obvious to me.
In the realm of vision, after a certain point you would just be trying to compress noise that has nothing to do with the content of image/video.
Lossless compression is a great objective because it's impossible to cheat. When you do lossy compression you have to define a quality metric and as soon as you do that the game becomes cheating the quality metric rather than actually compressing the data in a useful way.
I highly recommend watching the video I linked. Arithmetic encoding reduces the task of lossless compression to assigning probabilities for the next token, which is the exact task these models are trained to do.
http://prize.hutter1.net/