The thing that people are looking to for answers, the NN itself, does not have them. That's like looking to Newton's compass to understand his general law of gravitation.
The reason that LLMs trained on the internet and every ebook has the structure of human communication is because the dataset has that structure. Why does the data have that structure? this requires science, there is no explanation "in the compass".
NNs are statistical models trained on data -- drawing analogies to animals is a mystification that causes people's ability to think clearly he to jump out the window. No one compares stock price models to the human brain; no banking regulator says, "well your volatility estimates were off because your machines had the wrong thoughts". This is pseudoscience.
Animals are not statistical learning algorithms, so the reason that's uninformative is because it's false. Animals are in direct causal contact with the world and uncover its structure through interventional action and counterfactual reasoning. The structure of animal bodies, and the general learning strategies are well-known, and having nothing to do with LLMs/NNs.
The reason that I know "The cup is in my hand" is not because P("The cup is in my hand"|HistoricalTexts) > P(not "The cup is in my hand"|HistoricalTexts)
And there are a lot of people out there who do not a lot of reasoning.
After all optical illusions exist, our brain generalizes.
The same thing happens with words like the riddle about the doctor operating on a child were we discover that the doctor is actually a female.
And while llms only use text, we can already see how multimodal models become better, architecture gets better and hardware too.
I'm not interested that your best friend emits the same words in the same order as an LLM; i'm more interested that he does so because he enjoys you company whereas the LLM does not.
Engineer's overstep their mission when they assume that because you can substitute one thing for another, and sell a product in doing so, that this is informative. It isnt. I'm not interested in whether you can replace the sky for a skybox and have no one notice -- who cares? What might fool an ape is everything, and what that matters for science is nothing.
We don't just talk about a LLM we talk about a neuronal network architecture.
There is a direct link to us (neural networks)
I mostly agree with your points, but I still disagree with this premise. Humans (and other animals) absolutely are statistical reasoning machines. They're just advanced ones which can process more than "text" - they're multi-modal.
As a super dumb-simple set of examples: Think about the origin of the phrase "Cargo Cult" and similar religious activities - people will absolutely draw conclusions about the world based on their learned observations. Intellectual "reasoning" (science!) really just relies on more probabilities or correlations.
The reason you know the cup is in your hand is because P("I see a cup and a hand"|HistoryOfEyesight) + P("I feel a cylinder shape"|HistoryOfTactileFeeling) + .... > P(Inverse). You can pretend it's because humans are intelligent beings with deep reasoning skills (not trying to challenge your smarts here!), but humans learn through trial and error just like a NN with reinforcement learning.
Close your eyes and ask a person to randomly place either a cup from your kitchen in your hand or a different object. You can probably tell which one is it is. Why? Because you have learned what it feels like, and learned countless examples of cups that are different, from years of passive practice. Thats basically deep learning.
Having a body which changes over time that interacts with a world that changes over time makes animal learning not statistical (call it, say, experimental). That animals fall into skinner-box irrational behaviour can be modelled as a kind of statistical learning, but it actually isnt.
It's a failure of ecological salience mechanisms in regulating the "experimental learning" that animals engage in. Eg., with the cargo cults the reason they adopted that view was because their society had a "big man" value system based on material acquisition and western waring powers seemed Very Big and so were humiliating. In order to retain their status they adopted (apparently irrational) theories of how the world worked (gods, etc).
From the outside this process might seem statistical, but it's the opposite. Their value system made material wealth have a different causal salience which was useful in their original ecology (a small island with small resources), but it went haywire when faced with the whole world.
Eventually these mechanisms update with this new information, or the tribe dies off -- but what's going wrong here is that the very very non-statistical learning ends up describable that way.
This is indeed, why we should be very concerned about people skinner-boxing themsleves with LLMs
The "experiment" of life is what defines the statical values! Experimentation is just learning what the statistical output of something is.
If I hand you a few dice, you'd probably be able to guess the statistical probability of every number for given roll. Because you've learned that through years of observation building a mental model. If I hand you a weighted die, suddenly your mental model is gone, and you can re-learn experimentally by rolling it a bunch. How can you explain experimental learning except "statistically"?
> they adopted (apparently irrational) theories of how the world worked (gods, etc)
They can be wrong without being irrational. Building an airport doesn't make planes show up, but planes won't show up without an airport. If you're an island nation with little understanding of the global geopolitical environment of WWII, you'd have no idea why planes started showing up on your island, but they keep showing up, and only at an airport. It seems rational to assume they'd continue showing up to airports.
> that animals fall into skinner-box irrational behaviour can be modelled as a kind of statistical learning, but it actually isnt
What is it if not statistical?
Also skinner boxes are, in a way, perfectly rational. There's no way to understand the environment, and if pushing a button feeds you, then rationally you should push the button when hungry. Humans like to think we're smart because we've invented deductive reasoning, and we quote "correlation is not causation" that we're not just earning to predict the world around us from past experiences.
For almost everything in the world, action on it, changes it. There are vanishingly few areas where this isn't the case (most physics, most chemistry, etc.).
Imagine trying to do statistics but every time you sampled from reality the distribution of your sample changes not due to randomness, but because reality has changed. Now, can you do statistics? No.
It makes all the difference in the world to have a body and hold the thing you're studying. Statistics is trying to guess the shape of the ice cube from the puddle; animal learning is making ice cubes.
RL is doing just this, simulating an environment. And we can have an agent "learn" in that environment.
I think tying learning to a body is too restrictive. The
You strongly rely on the assumption that "something else" generates the statistics we observe, but scientifically, there exists little evidence whether that "something else" exists (see eg the Bayesian brain).
If you're just subject to time-varying, random, stochastic, perceptual distributions you have no way of estimating the properties of the data generating process (reality).
You need to be the one in control of the distribution in order to study it: this is the lesson of the whole history of science as an experimental discipline.
I'm always puzzled by such assertions. A cursory look at the technical aspects of an iterated attention - perceptron transformation clearly shows it's just a convoluted and powerful way to query the training data, a "fancy" Markov chain. The only rationality it can exhibit is that which is already embedded in the dataset. If trained on nonsensical data it would generate nonsense and if trained with a partially non-sensical dataset it will generate an average between truth and nonsense that maximizes some abstract algorithmic goal.
There is no knowledge generation going on, no rational examination of the dataset through the lens of an internal model of reality that allows the rejection of invalid premises. The intellectual food already chewed and digested in the form of the training weights, with the model just mechanically extracting the nutrients, as opposed to venturing in the outside world to hunt.
So if it works "just like the human brain", it does so in a very remote sense, just like a basic neural net works "just like the human brain", i.e individual biological neurons can be said to be somewhat similar.
Sure, we have a nice inner loop, we do some pruning, picking and choosing, updating, weighting things based on emotions, goals, etc.
Who knows how complicated those things will prove to model/implement...
Or if you just want to say that NNs are used as a statistical model here: Well, yea, but that doesn't really tell you anything. Everything can be a statistical model.
E.g., you could also say "this is exactly the way the human brain works", but it doesn't really tell you anything how it really works.