That said, LLMs are still trained on significantly more data pretty much no matter how you look at it. E.g. a blind child might hear 10-15 million words by age 6 vs. trillions for LLMs.
A camera hooked up to the baby's head is absolutely not getting all the input data the baby gets. It's not even getting most of it.
The acquired knowledge is a lot less uniform than you’re proposing and in fact is full of gaps a human would never make. And more critically, it is not able to peer into all of its vast knowledge at once, so with every prompt what you get is closer to an “instance of a human” than “all of humanity” as you might think of LLMs.
(I train and dissect LLMs for a living and for fun)
They mentioned the training data is much higher for an LLM, LLM's recall not being uniform was never in question.
No one expects compression to be without loss when you scale below knowledge entropy that exists in your training set.
I am not saying LLMs do simple compression but just pointing a mathematical certainity.
(And I think you don't need to be an expert in creating LLMs to understand them, albeit I think a lot of people here have experience with it aswell so I find the additional emphasis on it moot).
Again just my impression from exposure to many LLMs at various states of training (my last sentence was not an appeal to expertise)
I don't know how to count the amount of words a human encounters in their life, but it does seem plausible that LLMs deal with orders of magnitude more words. What I'm saying is that words aren't the whole picture.
Humans get continuous streams of video, audio, smell, location and other sensory data. Plus, you get data about your impact on the world and the world's impact on you: what happens when you move this thing? What happens when you touch some fire? LLMs don't have this yet, they only have abstract symbols (words, tokens).
So when I look at it from this "sensory" perspective, LLMs don't seem to be getting any data at all here.
I think this would depend entirely on how the sensory impairment came about, since most genetic problems are not isolated, but carry a bunch of other related problems (all of which can impact intelligence).
Lose your eye sight in an accident? I would grant there is likely no difference on average.
Otherwise, the null hypothesis is that intelligence (and a whole host of other problems) are likely worse, on average.
This is clearly untrue. All information a human ever receives is through sensory data. Unless your position is that the intelligence of a brain that was grown in a vat with no inputs would be equivalent to that of a normal person.
Now, does rotating a coffee mug and feeling its weight, seeing it from different angles, etc. improve intelligence? Actually, still yes, if your intelligence test happens to include questions like “is this a picture of a mug” or “which of these objects is closest in weight to a mug”.
Entirely possible - we just don’t know. The closest thing we have to a real world case study is Helen Keller and other people with significant sensory impairments, who are demonstrably unimpaired in a general cognitive sense, and in many cases more cognitively capable than the average unimpaired person.
This seems like a very philosophical standpoint, rather than practical. And I guess that’s fine, but I feel like the implication is that if an LLM is in some way intelligent, then it was exactly as intelligent before training. So we are talking about “potential intelligence“? Does a stack of GPU’s have “intelligence”?
I’m trying to illustrate that the constraints that apply to LLMs don’t necessarily apply to humans. I don’t believe human intelligence is reliant upon sensory input.
The term cannot mean something measurable or concrete when it’s convenient, but be vague and indefinable when it’s not.
Intelligence likely doesn’t require that much data, and it may be more a question of evolutionary chance. After all, human intelligence is largely (if not exclusively) the result of natural selection from random mutations, with a generation count that’s likely smaller than the number of training iterations of LLMs. We haven’t found a way yet to artificially develop a digital equivalent effectively, and the way we are training neural networks might actually be a dead end here.
Which gives us no information on computational complexity of running that algorithm, or on what it does exactly. Only that it's small.
LLMs don't get that algorithm, so they have to discover certain things the hard way.
Humans ship with all the priors evolution has managed to cram into them. LLMs have to rediscover all of it from scratch just by looking at an awful lot of data.
The fact that they can pull it off to this extent was a very surprising finding.
We get a lot of high quality data that's relatively the same. We run the same routines every day, doing more or less the same things, which makes us extremely reliable at what we do but not very worldly.
LLMs get the opposite: sparse, relatively low quality, low modality data that's extremely varied, so they have a much wider breadth of knowledge but they're pretty fragile in comparison since they get relatively little experience on each topic and usually no chance to affirm learning with RL.
I think it's the other way round: humans have effectively unbounded training data. We can count exactly how much text any given model saw during training. We know exactly how many images or video frames were used to train it, and so on. Can we count the amount of input humans receive?
I can look at my coffee mug from any angle I want, I can feel it in my hands, I can sniff it, lick it and fiddle with it as much as I want. What happens if I move it away from me? Can I turn it this way, can I lift it up? What does it feel like to drink from this cup? What does it feel like when someone else drinks from my cup? The LLM has no idea because it doesn't have access to sensory data and it can't manipulate real-life objects (yet).