Ie, what humans view as "intelligence" isn't actually an emergent property of the right set of rules. Rather, it's a massive and hopelessly self-intersecting set of ad-hoc solutions and special casing all jumbled together over millions of years of evolution, producing something that looks general, but only because we're looking at it from within itself. An easy example: Having a computer identify all images of a thing called a "cup". That's an arbitrary category that has no definition reducible to actual physical properties. What makes something a "cup"? A human saying it is one.
For something to look like "full, general AI" to humans, we'd need to either: Build something that replicates all that specialized hyper-meta-spaghetti-code mental processing of ours to a level of fidelity that's beyond both our current understanding of our brain's structure and the level of complexity tenable by human computer science. Or leave out those evolutionarily-discovered "optimization circuits" and require far, far more processing power than we'll have access to for a good, long time.
For example it is "easy" for a human to "see" that a coffee cup and a donut are topologically the same, whereas the same task for a computer, i.e. provide it with enough data to recognize something to have exactly one "hole" seems extremely hard. Another example would be to recognize that up to rotation two things are roughly the same. Clearly humans do not solve such problems case by case, they have some build in classifier system that solves those problems by a general principle.
And the human got their definition of cup by seeing a lot of things others called a cup and extrapolating rules, something that's already obtainable for some problems with solutions like neural networks.
The problem is: nobody knows. Its a bit like two cavemen banging rocks together until one says "hey, lets use these rocks to kill something" and the other one goes "eh? How?" and the first one says "I don't know, keep bashing until something dies.."
(Diagnostic experiment: Is there a significant subset of "cups" that a naive neural network would always fail to include or only include when also including a significant subset of non-"cups"?)
But nobody was arguing about the infeasibility of approximations of specific human cognitive abilities. Wasn't the conversation about general human cognitive ability?
Google is doing some research. One quote:
"One of my favourite things is artificial intelligence, but it has gotten a very bad rap, but my prediction is that when AI happens it's going to be a lot of computation and not so much clever algorithms but just a lot of computation. My theory is that if you look at your programming, your DNA, it's about 600 megabytes compressed, so it's smaller than any modern operating system, smaller than Linux or Windows or anything like that, your whole operating system, that includes booting up your brain. So your program algorithms probably aren't that complicated, it's probably more about the overall computation. We have some people at Google who are trying to build artificial intelligence and to do it on a large scale to make search better. Very few [other] people are working on this, and I don't think it's as far off as people think." - Larry Page, February 2007
http://www.abc.net.au/radionational/programs/scienceshow/cha...
http://en.wikipedia.org/wiki/Hybrot
edit - as it gets more sophisticated it is going to generate some fairly strange debates. What rights do you give to a bathtub of human neurons that you are trying to use to predict the stock market?
My childhood passed in reading circa-computer books of 60s, 70s, 80s, where full, general AI seemed to be just around the corner. Obviously this problem is far harder than it seemed to be.
But the (almost universal) lack of trying is extremely disheartening.