"Specialized at being human" - this is a deep intuition. We are reinforcement learning agents that are pre-programed with a certain number of reward responses. We learn from rewards to keep ourselves alive, to find food, company and make babies. It's all a self reinforcing loop, where intelligence has the role of keeping the body alive, and the body has the role of expressing that intelligence. We're really specialized in keeping human bodies alive and making more human bodies, in our present environment.
The author puts a hard limit on intelligence because intelligence is limited by the complexity of the problems it needs to solve (assuming it has sufficient abilities). So the environment is the bottleneck. In that case, an AGI would be like an intelligent human, a little bit better than the rest, not millions of times better.
If it were valid, one could take the argument in your last paragraph to draw a line at any point in the evolution of intelligence and say "this is as good as it gets."
But the question is not, "is intelligence explosion possible?" The question is, "explode into what?"
I think there are two usages of the term 'general intelligence' floating around:
(1) the ability that humans possess (but which animals don't) to create universal theories,
(2) the measure of one human's general cognitive ability or potential (in all fields) relative to another human's.
Note that IQ tests are concerned with (2). The quest for AGI is concerned with (1), though the additional prediction of intelligence explosion or singularity assumes the validity of (2).
I think the author would claim that (1) exists but (2) doesn't. He explains the predictive power of IQ tests by claiming that general intelligence is a threshold ability and that people who score highly on an (arbitrary) test are more likely to have exceeded that threshold. Beyond the threshold, achievement is limited only by other factors.
But, regardless of this, I think solving problems requires creativity, not intelligence. Creativity seems to be independent of knowing how to do IQ tests. It's also, pace the author, independent of the environment. The main limiting factor is whether you want some particular knowledge.
>sufficient number of communicating, above-average intelligent agents could expand the intellectual environment
Haven't we already got that, with the internet?
Groups tend to be dominated by groupthink which is why creative individuals are aloof.
Nevertheless, a young group of AGIs would be fairly isolated from humanity simply by virtue of being non-human. So, though their starting point would be determined by the state of our knowledge at their birth, they may well make rapid progress for a short period, rather like the renaissance or the USA in the late 1800s when the nation was young and expanding. Then presumably they would fall prey to groupthink and pessimism just as most adults and nations do eventually. Progress would slow considerably.
>>sufficient number of communicating, above-average intelligent agents could expand the intellectual environment
>Haven't we already got that, with the internet?
Up to a point (with all the intelligent agents being human, as we don't have AGI), but it would be rather surprising if what we see now just happens to be a hard upper limit.
I am not necessarily convinced by all of the points you raise, but they are certainly reasonable in the context of arguing that singularity-like events are by no means inevitable (a point of view that I share). What the author is claiming, however, is that it is impossible, which means that he has imposed on himself a burden of proof (that goes beyond plausibility) for a number of conjectures.
It's not a tautology. "Generality" and "specificity" are artifacts of the human experience. What is tautological is to say, "it's general to me, therefore it is general."
You think that more and more progress comes by way of more and more optimization. This is not the whole story. Accidentation is the missing ingredient. Humans - as well as all life on earth - have a knack for creating more and more problems. It is this never-ending fountain of new, accidental problems that allows for what appears to us to be a chain of "progress" stretching into the past.
Our "generality" is in fact a hairball collection of specific functions that have accreted into the human animal over millions and millions of years. Some abstract Java class called `Agent` with an `.optimize()` method hanging off of it simply does not have that context.
If you want a really high quality, generally intelligent function in silicon though, it's hard to beat the XOR function ;)
I think the author just means "the skills needed to pilot a human body on Earth in a normal human social environment". I see no tautology, and the preceding sentence about octopuses makes the author's meaning here pretty clear.
Oh wait. Those do not fall under such narrow definition of intelligence.
No reason why a GAI wouldn't have a similar instinct or capabilities.
Author also misposes no free lunch theorem. It pertains to algorithms not computational ability, and we're entirely not sure if true intelligence is an immutable algorithm. (Human is not, because of genetics.) There is no real equivalent theorem for self modifying algorithms, (or changing objective functions) and if there were, it will run afoul of halting problem.
Three theorem also had at least one known hole where it does not apply to coevolutionary algorithms. (Such as part of Alpha Go to go for an easy example.)
While environment puts on a hard limit, the limit is extremely high, on the order of 2^300. The optimising does not have to go for the simplest algorithm to solve the problem. And we do not know enough about informational capacity of matter in quantum states to be honest.
How do we know that? We only know of intelligences that co-evolved with their sensoriotor etc, so how do we know that's not the only way to do it?
It starts by claiming that there is no such thing as general intelligence. What specialized intelligence, then, is human intelligence? It's specialized for "being human". The author is apparently unaware that this tautological response eliminates the distinction between general and specialized intelligence, as one could just as validly (or vacuously) say that a superhuman intelligence is specialized in being what it is and doing what it does. The author has invalidated the hook on which he had hung his argument.
A lot of column-inches are expended on repeatedly restating that animal intelligences have co-evolved with their sensorimotor systems, which is a contingent fact of history, not a fundamental necessity for intelligence (as far as we know; but then the whole article is predicated on the feasibility of AI.) He raises the 'brain in a vat' trope, but no-one is suggesting that AIs must be disconnected from the external world. Furthermore, this line of argument ignores the fact that many of the greatest achievements of human intelligence have come from contemplating abstract ideas.
When the author writes "most of our intelligence is not in our brain, it is externalized as our civilization", he is confusing the achievements of intelligent agents for intelligence itself. When he writes that "an individual brain cannot implement recursive intelligence augmentation" he is confusing a limit on human capabilities for a fundamental limit on intelligence itself...
I am far from convinced that the singularity must follow from achieving human-level artificial intelligence, as we don't know how to get to the starting line, let alone know how the problem of bootstrapping intelligence scales, but the arguments presented here do nothing to persuade me that it is impossible.