News for hackers. Stuff that matters.
Re:Compression Is Prediction by TeMPOraL on Tuesday August 11, 2026 @06:06PM (#49265141)
And yet to this day, in AI threads, so many people act shocked and surprised if you dare follow the obvious implication and claim that understanding is a form of lossy compression.
Re:Compression Is Prediction by kazinator on Tuesday August 11, 2026 @06:36PM (#49265438)
Shorter description isn't understanding, let alone of it is lossy.

When you shorten a description in a lossy way, you are deciding a priori that some differences in the object don't matter, and it's not because you understand the object, but because it serves your goal of shortening the description.

Re:Compression Is Prediction by Dylan16807 on Tuesday August 11, 2026 @07:09PM (#49265730)
Or you actually do understand it.

You can't just assume smaller is better but it often is. And very often it's more information-dense.

Re:Compression Is Prediction by kazinator on Tuesday August 11, 2026 @09:40PM (#49266886)
You can compress syntax, losslessly even, with zero understanding of its semantics. Zero understanding not only imbued into the compressor/decompressor, but even the designer of the compressor doesn't require understanding the semantics. Actually, even of the syntax.

A compression program can compress a book written in a language that the author of the program doesn't understand, on a topic he knows little about.

Re:Compression Is Prediction by Dylan16807 on Tuesday August 11, 2026 @10:18PM (#49267133)
Finding common characters and building a list of words is a low level type of understanding. Doing it better does actually start directly representing syntax patterns and that's a less-low level of understanding.

I think "losslessly even" is the wrong way to think about it. Lossless compression often requires less understanding than high quality lossy compression. If you can do a lossy compression that correctly decides what details are unimportant, that's a good sign of understanding.

Re:Compression Is Prediction by skinfaxi on Tuesday August 11, 2026 @10:50PM (#49267309)
> I think "losslessly even" is the wrong way to think about it. Lossless compression often requires less understanding than high quality lossy compression. If you can do a lossy compression that correctly decides what details are unimportant, that's a good sign of understanding.

This is the crux and reminds me of things like mp3 that exploit the nature of human hearing being limited to a frequency range.

Re:Compression Is Prediction by vintermann on Wednesday August 12, 2026 @01:48AM (#49268264)
We model the data. The model, hopefully, captures something real in the data. If it does, then it's fair to say that we understand the data better.

But it's frankly a philosophical question what's real or not. No model is going to capture absolutely everything about the thing it models - at that point, it would be the thing. The best we can hope for is that it captures everything we care about.

And no experiment or metric can tell you if you care about the right things. At best it can tell us if we care about a thing given other things we care about. "No cares in, no cares out".

To make it a little more concrete: you could compress a string from back to front. You could build an LLM to help you do that. If you care about file size, that's almost certainly a bad idea, the forward LLM will be better for that purpose. But are there purposes for which the backward LLM might be better? I think that's not so hard to imagine. Often we wonder about "what came before".

Re:Compression Is Prediction by Dylan16807 on Wednesday August 12, 2026 @03:41AM (#49269018)
A model will not capture every nuance of what it models, but it might capture every nuance you managed to measure properly.
Re:Compression Is Prediction by TeMPOraL on Wednesday August 12, 2026 @04:35AM (#49269389)
Including the things you didn't even realize you measured, or even conceptualized at all.
Re:Compression Is Prediction by adrianN on Tuesday August 11, 2026 @11:42PM (#49267606)
I'm not aware of a better definition of "understanding" that would allow me to tell whether some system "understands" some other system. Do you happen to know one?
Re:Compression Is Prediction by hnfong on Wednesday August 12, 2026 @01:45AM (#49268232)
See: A. M. Turing (1950) Computing Machinery and Intelligence. Mind 49: 433-460.

I mean, my interpretation is that the question Turing tried to answer is equivalent to "How can we determine whether machines understand humans/human thought?"

This only works when both systems can talk about pretty much arbitrary things, but if you want a more general method for less complex systems, perhaps having one system simulate another system is sufficient. (Which is also another Turing invention)

Re:Compression Is Prediction by adrianN on Wednesday August 12, 2026 @03:06AM (#49268783)
There are many people who would claim that passing the Turing test is insufficient to show "understanding" (compare for example the Chinese Room thought experiment).
Re:Compression Is Prediction by hnfong on Wednesday August 12, 2026 @05:54AM (#49269932)
Yes but it's (kind of?) a definition as you asked for.

At this point, I am unaware of a better definition. I know the Chinese Room argument (and I disagree with it), but I'm not aware whether the proponents of that argument have a better definition of understanding other than "well, the Turing Test isn't enough"...

---

PS: Interestingly the issue of compression is highly relevant regarding the Chinese Room argument -- the essential element in the Chinese Room argument is that the information is not compressed...

Re:Compression Is Prediction by ablob on Tuesday August 11, 2026 @06:37PM (#49265447)
If you can predict other things by understanding something, is it truly lossy compression?
Re:Compression Is Prediction by bonoboTP on Tuesday August 11, 2026 @08:45PM (#49266464)
Of course it is but again "X is just Y" is often used to mislead. A brain is just neurons! A computer is just transistors! An LLM just predicts the next token! It's just like a parrot! It's just like a blurry jpeg of the internet! Kinda yes, but what do you use this for? It's a bad intuition pump is it leads people to conclude demonstrably false things about capabilities.
Re:Compression Is Prediction by qsera on Tuesday August 11, 2026 @11:55PM (#49267665)
>understanding is a form of lossy compression..

No. Teaching is compression. Understanding and application is decompression.

Re:Compression Is Prediction by Dylan16807 on Wednesday August 12, 2026 @12:08AM (#49267729)
I would roughly say that understanding is compression and application is decompression.

You don't memorize the specific details, you memorize the flow of systems. But you get back to the details to put things into practice.

Re:Compression Is Prediction by qsera on Wednesday August 12, 2026 @01:03AM (#49267992)
I was speaking in the context of humans. When someone teaches you, the content coming from the teacher is very compressed. One decompress it when they can generalize and apply it. So understanding is compressed, but is not the act of compressing. I mean it is not compressed from a larger data or made by compressing a larger data. The larger data it represents never existed. It is like the definition of a fractal...
Re:Compression Is Prediction by Dylan16807 on Wednesday August 12, 2026 @01:31AM (#49268148)
> I was speaking in the context of humans.

So was I.

> One decompress it when they can generalize and apply it.

Generalizing is compression. Generalizing and applying are very different things.

> So understanding is compressed, but is not the act of compressing.

Understanding (noun) is compressed. Understanding (verb) is the act of creating understanding (noun) and is compression.

> The larger data it represents never existed.

The larger data is everything you take in about the subject. Experience and instruction.

Re:Compression Is Prediction by qsera on Wednesday August 12, 2026 @02:07AM (#49268394)
>creating understanding (noun) and is compression.

What is it compressing from?

Re:Compression Is Prediction by WithinReason on Wednesday August 12, 2026 @03:20AM (#49268886)
examples
Re:Compression Is Prediction by vrighter on Friday August 14, 2026 @07:36AM (#49297433)
I think of it as more "providing you with the model you can use to decompress". You only really learn something when you do it yourself, not just listen to someone talk about it.