A nice source for this might be "The unbearable slowness of being: Why do we live at 10 bits/s?" [1].

A lot of people seem to think that training is limited to the 10 bits/s of processing mentioned here. But raw sensor input (mostly visual) is in the order of 21 quadrillion bits per year, according to the same paper. It seems likely to me that learning uses much more data than is ultimately expressed in behaviour.

[1] https://www.sciencedirect.com/science/article/pii/S089662732...

That "raw sensor input" is massively, hideously spatially and temporally redundant.

Even the shallow encoding patterns of AV1 can easily crush a raw video stream down by a factor of 10^3 without losing either spatial coherence or semantic loading.

Humans live in self-similar environments and perform self-similar activities. Any practical training dataset has the "natural human data" beat on diversity by a crushing amount. And a system that can learn deeper patterns makes the sensor streams more redundant, not less.

If it weren't massively redundant (ie identifiable patterns) there would have been no point evolving brains to parse the raw sensor input.

Nah. The "huge amounts of data" humans receive are incredibly self-redundant.

Incorporating non-text data into AI training runs has repeatedly demonstrated that for general reasoning capabilities, no data type beats text in value per byte.

It is very redundant, but there is an incredible amount of it and we can "train" continuously and much more efficiently.

This is only true on the surface. Yes, making a model a good debator can be acchieved like this, but every current frontier model exhibits the same semantc confusion when it comes to actual mechanics of physical objects. Understanding how not to pass the ball through the glass is not a property of textual reasoning. This is also very visible in model generated code where the ball and the glass plane very often intersect. Easy to identify in syntax (compilers will do the job for you), easy to miss on the concept level.

This becomes more true the deeper you look.

LLMs used to "fail at spatial reasoning forever". But each new generation would fail a little less. Then we got the likes of Astra, and ARC-AGI-3 straight up died.

Learning something from a lecture, seminar or a book is close equivalent to learning from text/tokens.