Re:Noam Chomsky on Where Artificial Intelligence Went Wrong
by antics on Monday January 21, 2013 @01:20PM
(#5092796)
Chomsky's argument here is basically the same argument he used to reinvent linguistics in the 50's and 60's, so understanding what's really being said here depends on a bit of context. Since neither previous discussions nor this article has quite nailed it, the discussion is indeed worth having (sorry, ColinWright).
The crux of the debate is this. When Chomsky was a grad student at UPenn most linguists thought that language was learned by a complicated mimicry -- that we learn language by imitating behavior, similar to how birds learn to call. The "hard problems" of linguistics were completely solved and linguistics had become a primarily classificatory science, with linguists simply cataloging words into parts of speech. This line of thought was interchangeably known as behavioralism or empiricism.
One of Chomsky's transformative insights was that most sentences that are reasonably long are completely unique in human history, and will also never again be uttered by another person, ever. (For example, try Googling that exact sentence.) One consequence of this is that people realized that the mimicry argument could not really account for the robust well-formed structure of sentences and give us an infinite set of them. What we need to generate an infinite set of well-structured sentences is a grammar. Thus the universal grammar was born, and while Chomsky did not convince all the prominent linguists of the time, he did convince all their grad students, and the field of linguistics has seldom looked back.
Where this begins to intersect with AI is where Chomsky is usually criticized for not having been quite revolutionary enough. His outline of linguistics basically split the field into semantics (which studies the meaning of language) and syntax (which studies the structure of language). He argued that everything about language that must be interpreted (like meaning) must go on the semantics side of the line, and everything else should go on the other side of the line. He does not believe syntax to be interpretive at all, and tends to react violently when anyone tries to push empiricism into the syntax dialogue. Even a lot of his students don't buy that syntax is completely not-interpretive, and so someone like Lakoff would claim that if he was revolutionary, he was not quite revolutionary enough.
Here's what this means for AI. Chomsky sees the statistical approach to learning as a type of empiricism. You take a corpus, learn some stuff statistically, and then perform well on a task. To someone like Chomsky this probably looks like Skinner's old model, but instead of words like "mimicry", we use words like "statistical inference." Remember that empiricism and syntax should be strictly separate, and it becomes easy to see why something like this would make him cranky.
Of course, computer scientists like Norvig and linguists like Lakoff disagree. Their collective argument is that some aspects of syntax are indeed interpretive, and that (in the case of Norvig) they can be learned statistically (using, e.g., PCFGs). For example consider the sentence "John called Mary a Republican and then SHE insulted HIM". This really only makes sense if you presume that the participants think that "Republican" is an insult, but how do you know that? The answer seems to be through some sort of past experience, which Norvig would say can be and should be modeled statistically. And that in short is the debate and why it exists.
The other complaints with what Chomsky said here are that it's scientifically incorrect. Chomsky claimed, for example, that statistical models are basically not real science, and not used in the history of science, which is obviously wrong. Norvig pointed out, for example, that in physics sometimes our only choice is to infer something statistically, as in the case of the gravitational constant or the Higgs boson. But given Chomsky's history, it's fair to assume he meant this in the context of behavioral science, in which case his point is mostly true (modulo the "old" model of linguistics, which he hates). But of course it's important to remember that his model of linguistics was also without precedent in the history of science, so that alone is not really justification for his position.
2. > One consequence of this is that people realized that the mimicry argument could not really account for the robust well-formed structure of sentences and give us an infinite set of them.
What about gradient well-formedness? Does anyone even in generative linguistics believe in programming language-hard grammaticality? What about acceptability?
3. > What we need to generate an infinite set of well-structured sentences is a grammar. Thus the universal grammar was born ... the field of linguistics has seldom looked back.
...which grammarless statistical models have no problem doing (or if you want to reject them for making mistakes, do you reject people?)
Which linguistic univerals, if any, have survived the test of time? What are some testable predictions it has made?
I hope I'm not missing the point, and apologies if this has already been said (but not in these exact words, ha ha...)
Re:Noam Chomsky on Where Artificial Intelligence Went Wrong
by david927 on Monday January 21, 2013 @11:22AM
(#5092098)
Chomsky on the lack of progress in AI:
If you take a look at the progress of science, the sciences are kind of a continuum, but they're broken up into fields. The greatest progress is in the sciences that study the simplest systems. So take, say physics -- greatest progress there. But one of the reasons is that the physicists have an advantage that no other branch of sciences has. If something gets too complicated, they hand it to someone else.
If a molecule is too big, you give it to the chemists. The chemists, for them, if the molecule is too big or the system gets too big, you give it to the biologists. And if it gets too big for them, they give it to the psychologists, and finally it ends up in the hands of the literary critic, and so on.
Re:Noam Chomsky on Where Artificial Intelligence Went Wrong
by cliffbean on Monday January 21, 2013 @01:08PM
(#5092735)
I find your finding of insight in this quote to say more about the
challenge of AI than the quote itself ;-).
First, the quote directly contradicts itself. The first paragraph says that physicists alone hand their hard problems to others. The second describes how the chemists, biologists, and psychologists all also hand their hard problems to others.
Second, it ignores how nearly all fields of human endeavor hand of various kinds of hard problems to other fields. Software engineers hand off many hard problems to the mathematicians, philosophers, artists, hardware engineers, and sometimes even to other software engineers in adjacent levels in the software stack ;-). Similar relationships exist between archaeologists, anthropologists, paleontologists, geologists, climatologists, meteorologists, and so on. It's easy to think of examples.
Geologists can't hand all of their hard problems off to climatologists, but physicists can't hand all of their hard problems off to chemists either.
And yet, somehow, humans are able to perceive meaning from quotes like this. Our ability to read a quote, ignore what it actually says and figure out what we think it was meant to say is truly stunning.
Re:Noam Chomsky on Where Artificial Intelligence Went Wrong
by agumonkey on Monday January 21, 2013 @11:56AM
(#5092278)
I watched 'The manufacture of consent' recently, I had a hard time stopping listening to him. His wording dug right through my skull, very concise yet simple.
Re:Noam Chomsky on Where Artificial Intelligence Went Wrong
by bones6 on Monday January 21, 2013 @01:03PM
(#5092715)
He's the most concise when it comes to explaining complicated ideas. Einstein said something about simple explanations but not too simple. The biggest point Chomsky made to me was that no real explanation or discussion can be had when you are limited to 2 minute soundbites on TV. I won't give his example but he said "You can't say <extremely flammatory counterculture yet objectively true statement> and not take the time to explain why."
Re:Noam Chomsky on Where Artificial Intelligence Went Wrong
by numlocked on Monday January 21, 2013 @11:56AM
(#5092280)
As someone who was familiar with the differences between Chomsky and Norvig (a simplification - maybe better to say Chomsky's view that developing empirical models is not scientifically interesting), I found this really enriched my understanding of Chomsky's view and gave me a lot of subtlety. He certainly doesn't seem antagonistic towards statistic modeling the way I imagined; just sort of bored by it. Be sure to persist to the end - there are some fun word games on the final page.
Re:Noam Chomsky on Where Artificial Intelligence Went Wrong
by saosebastiao on Monday January 21, 2013 @12:43PM
(#5092595)
Norvig still wins, IMO. Statistics may not explain everything, but it does an amazing job of separating the explainable from the unexplainable. I still see this as the requisite Minimum Viable Product for Artificial Intelligence. If we sat around waiting to perfectly understand neurocognitive linguistic processes before we tried to replicate it, we wouldn't have a functioning Google Translate for another 200 years.
The crux of the debate is this. When Chomsky was a grad student at UPenn most linguists thought that language was learned by a complicated mimicry -- that we learn language by imitating behavior, similar to how birds learn to call. The "hard problems" of linguistics were completely solved and linguistics had become a primarily classificatory science, with linguists simply cataloging words into parts of speech. This line of thought was interchangeably known as behavioralism or empiricism.
One of Chomsky's transformative insights was that most sentences that are reasonably long are completely unique in human history, and will also never again be uttered by another person, ever. (For example, try Googling that exact sentence.) One consequence of this is that people realized that the mimicry argument could not really account for the robust well-formed structure of sentences and give us an infinite set of them. What we need to generate an infinite set of well-structured sentences is a grammar. Thus the universal grammar was born, and while Chomsky did not convince all the prominent linguists of the time, he did convince all their grad students, and the field of linguistics has seldom looked back.
Where this begins to intersect with AI is where Chomsky is usually criticized for not having been quite revolutionary enough. His outline of linguistics basically split the field into semantics (which studies the meaning of language) and syntax (which studies the structure of language). He argued that everything about language that must be interpreted (like meaning) must go on the semantics side of the line, and everything else should go on the other side of the line. He does not believe syntax to be interpretive at all, and tends to react violently when anyone tries to push empiricism into the syntax dialogue. Even a lot of his students don't buy that syntax is completely not-interpretive, and so someone like Lakoff would claim that if he was revolutionary, he was not quite revolutionary enough.
Here's what this means for AI. Chomsky sees the statistical approach to learning as a type of empiricism. You take a corpus, learn some stuff statistically, and then perform well on a task. To someone like Chomsky this probably looks like Skinner's old model, but instead of words like "mimicry", we use words like "statistical inference." Remember that empiricism and syntax should be strictly separate, and it becomes easy to see why something like this would make him cranky.
Of course, computer scientists like Norvig and linguists like Lakoff disagree. Their collective argument is that some aspects of syntax are indeed interpretive, and that (in the case of Norvig) they can be learned statistically (using, e.g., PCFGs). For example consider the sentence "John called Mary a Republican and then SHE insulted HIM". This really only makes sense if you presume that the participants think that "Republican" is an insult, but how do you know that? The answer seems to be through some sort of past experience, which Norvig would say can be and should be modeled statistically. And that in short is the debate and why it exists.
The other complaints with what Chomsky said here are that it's scientifically incorrect. Chomsky claimed, for example, that statistical models are basically not real science, and not used in the history of science, which is obviously wrong. Norvig pointed out, for example, that in physics sometimes our only choice is to infer something statistically, as in the case of the gravitational constant or the Higgs boson. But given Chomsky's history, it's fair to assume he meant this in the context of behavioral science, in which case his point is mostly true (modulo the "old" model of linguistics, which he hates). But of course it's important to remember that his model of linguistics was also without precedent in the history of science, so that alone is not really justification for his position.
1. > One of Chomsky's transformative insights was that most sentences that are reasonably long are completely unique in human history
... as with a sufficient amount of information in every error-tolerant format.
Every photo ever taken of the Mona Lisa is unique because the pixels are different, but has a very similar high-level meaning, echoing the paraphrase detection task results discussed in http://www.socher.org/index.php/DeepLearningTutorial/DeepLea....
2. > One consequence of this is that people realized that the mimicry argument could not really account for the robust well-formed structure of sentences and give us an infinite set of them.
What about gradient well-formedness? Does anyone even in generative linguistics believe in programming language-hard grammaticality? What about acceptability?
3. > What we need to generate an infinite set of well-structured sentences is a grammar. Thus the universal grammar was born ... the field of linguistics has seldom looked back.
...which grammarless statistical models have no problem doing (or if you want to reject them for making mistakes, do you reject people?)
Which linguistic univerals, if any, have survived the test of time? What are some testable predictions it has made?
I hope I'm not missing the point, and apologies if this has already been said (but not in these exact words, ha ha...)
If you take a look at the progress of science, the sciences are kind of a continuum, but they're broken up into fields. The greatest progress is in the sciences that study the simplest systems. So take, say physics -- greatest progress there. But one of the reasons is that the physicists have an advantage that no other branch of sciences has. If something gets too complicated, they hand it to someone else.
If a molecule is too big, you give it to the chemists. The chemists, for them, if the molecule is too big or the system gets too big, you give it to the biologists. And if it gets too big for them, they give it to the psychologists, and finally it ends up in the hands of the literary critic, and so on.
First, the quote directly contradicts itself. The first paragraph says that physicists alone hand their hard problems to others. The second describes how the chemists, biologists, and psychologists all also hand their hard problems to others.
Second, it ignores how nearly all fields of human endeavor hand of various kinds of hard problems to other fields. Software engineers hand off many hard problems to the mathematicians, philosophers, artists, hardware engineers, and sometimes even to other software engineers in adjacent levels in the software stack ;-). Similar relationships exist between archaeologists, anthropologists, paleontologists, geologists, climatologists, meteorologists, and so on. It's easy to think of examples.
Geologists can't hand all of their hard problems off to climatologists, but physicists can't hand all of their hard problems off to chemists either.
And yet, somehow, humans are able to perceive meaning from quotes like this. Our ability to read a quote, ignore what it actually says and figure out what we think it was meant to say is truly stunning.
* http://news.ycombinator.com/item?id=4729068
That one even gave the link to the single page version:
* http://www.theatlantic.com/technology/archive/2012/11/noam-c...
Many other submissions which I won't enumerate
I've found chomsky, n (2005). Three factors in language design, but was wondering if anyone knew of anything else, or anything better.