That's just not true. There's no more super cheap data, but there's still probably about 5x the data we are currently feeding these models just waiting to be scanned and OCRd, and most of it is higher quality (if less topical).

The fact that AI requires so much data does make me think that we are somehow on the wrong track. Evolution didn't scan every atom on earth in order to produce a brain. FSD is a good example of this. It is ridiculous to me that we need to show the car videos of every possible situation in order for it to know how to drive.

It’s not the wrong track, it’s just very early and specialized. As you get more multi-modal models with better ability to learn/remember on the fly, they will require less and less data to train.

I'm pretty sure we will look back and laugh about how this was done in 50 years. No disrespect to anyone working in the area now of course, but I think better, more natural approaches will emerge.

> The fact that AI requires so much data does make me think that we are somehow on the wrong track.

I know the entire AI industry on the wrong track, but there's nothing that can be done about it. Sometimes you just have to wait for the crash and burn, and then things can change. It's like being at the ocean and trying to stop a wave from crashing after it has already formed, you just have to let it resolve itself of its own volition. As Einstein said, stupidity is infinite.

Not just that. Language acquisition. Teacher never stands in front of class and says "here's what the rules of grammar aren't. You today learn want do not how." There is a paucity of negative examples in the "training". Same goes for so many other tasks we learn quicker than can be explained with a Hebbian punishment/reward model. Most people can play a board game somewhat competently after just being told the rules. Some are even good at it. You don't need to crash your car dozens of time to figure out how to drive. All over we see something fundamental missing, and the industry is trying to hammer the screw into the wall while pretending that's the best way to do it.