Re:On Chomsky and the Two Cultures of Statistical Learning
by jackmott42 on Thursday March 09, 2023 @11:28AM
(#35083049)
I have a lot of hype because of how capable these systems are with what is essentially a naive and simple approach. (granted with TONS of data thrown at it).
Adding in layers of complexity is going to happen next, and I expect it is going to get wild.
Re:On Chomsky and the Two Cultures of Statistical Learning
by ChatGTP on Thursday March 09, 2023 @08:37PM
(#35089382)
We’ll that’s assuming adding in layers of complexity yields fruitful. I can think of a lot of systems that don’t necessarily benefit from being more complex.
Re:On Chomsky and the Two Cultures of Statistical Learning
by dragonwriter on Thursday March 09, 2023 @12:17PM
(#35083687)
> Had he said "not ONLY a statistical engine for pattern matching," I would agree.
I’m not sure I would even then. I mean maybe, but that’s not the clearest, most certain, fundamental difference.
A more clear fundamental difference is ChatGPT instances are all, metaphorically, instinct with no space for intelligence. They have lots of “training”, but that all happens before they are capable of acting, and once they are capable of acting their behavior is entirely preprogrammed based on a very small input window. They have no memory of experience (that’s simulated, within the token limit, between the model and the user for chatbots), much less a reward mechanism that would let them learn behavior from experience.
Had he said "not ONLY a statistical engine for pattern matching," I would agree. But I'm pretty sure that the majority of what human cognition is and does is exactly what he describes. The pattern matching engine encodes the wisdom of thousands of years of evolution, in addition to the experiences of any particular instance (person), but in the end, most of what we do and say is generated by predicting, based on pattern matching, what is most likely to get the brain positive feedback.
The thing that the rush to more and bigger LLMs seems to leave out, to me, is that the pattern matching and prediction that drives intelligent response and behavior in humans (when you can find it), is not merely prediction about language, but rather prediction based on multiple learned (by humanity generally through evolutionary incorporation into our neural architecture, and by individuals through experience) models - of space, time, biology, abstract reasoning, ontologies that objectify the natural and human intellectual worlds, and more. LLMs encode very little if any of that directly, but rather get pieces of it indirectly through the imprint of ontology and reason that are baked into word (token) usage patterns.
So, I agree with Chomsky on the main point: you can't really get much beyond fluency with LLMs alone. There is way too much hype on these things.