Re:Quadrillion-parameter transformers and how to train them
by ACCount39 on Sunday November 08, 2026 @10:30AM
(#50414101)
That "raw sensor input" is massively, hideously spatially and temporally redundant.
Even the shallow encoding patterns of AV1 can easily crush a raw video stream down by a factor of 10^3 without losing either spatial coherence or semantic loading.
Humans live in self-similar environments and perform self-similar activities. Any practical training dataset has the "natural human data" beat on diversity by a crushing amount. And a system that can learn deeper patterns makes the sensor streams more redundant, not less.
A lot of people seem to think that training is limited to the 10 bits/s of processing mentioned here. But raw sensor input (mostly visual) is in the order of 21 quadrillion bits per year, according to the same paper. It seems likely to me that learning uses much more data than is ultimately expressed in behaviour.
[1] https://www.sciencedirect.com/science/article/pii/S089662732...