The author is quite explicit about this: he is contradicting that the singularity must be real, not trying to claim that the singularity can't be real. If you agree with his refutation of the arguments for the singularity, you'll be left with the claim "the singularity may or may not happen, we don't know", not with the claim "it's impossible for the singularity to happen".
> > This ability of humans to build on one another’s work is precisely why I don’t believe that running a human-equivalent A.I. program for a hundred years in isolation is a good way to produce major breakthroughs.
> As the saying goes, if I'm going to spend ten hours cutting down trees, I'd spend nine of them sharpening my axe. Just as a society of humans can develop tools to improve their capabilities, so can a single human, as the author's own example of Isaac Newton shows. If the AI is going to be locked in isolation for a hundred years to write software, why doesn't the author think it will spend some of that time on tools to improve its own productivity?
I think a better argument against the author's claim here is that thinking of 1 AI as 1 human is probably wrong. Since an AI can be trivially copied and run independently, it's more useful to think of 1 AI as 1 society.
We can, however, create faster computers. This is the second or third argument, and here the author gets something fundamentally wrong. We have, for the past decades, used faster computers to help us make faster computers. Their argument is that it's self-limiting, but that's provably false viz current CPU architectures.
The one thing the author skirts around that DOES seem to be a strike against the singularity is the asymptotic nature of those kinds of improvements. We can spend N amount of resources for a 100 unit improvement in speed/efficiency/whatnot, but spending 10N resources does not give us 1000 unit improvement, but rather 500 or 150.
Suppose there were a group of monks dedicated to achieving high intellect. One could imagine that they form a pact for many generations that only the most intelligent among them will have children. It's hard to imagine that this wouldn't result in intelligence increases.
Human brains fire only paths in use (and do everything in parallel), but a neural network needs to recalculate the weights on the whole network each training pass for example. Maybe there are some newer techniques that reduce that somewhat.
I think it's not a load bearing assumption for the intelligence explosion hypothesis, but just a premise that allows the notion of "more intelligent than" to be meaningfully expressed, for the sake of having a conversation about this idea. (Comparisons imply some way of ordering things, even if those things are nebulous and multidimensional concepts).
> For example, it’s entirely possible that the best that a person with an I.Q. of 300 can do is increase another person’s I.Q. to 200. That would allow one person with an I.Q. of 300 to grant everyone around them an I.Q. of 200, which frankly would be an amazing accomplishment. But that would still leave us at a plateau; there would be no recursive self-improvement and no intelligence explosion.
When it's phrased in terms of increasing the intelligence of already-living people, that makes it sound harder than when you phrase it as building a new thing that's better at building than the thing that built it. We can, for example, make machine tools that are faster or more accurate than the machine tools we make them with.
I think the premise of an intelligence explosion usually starts with human programmers creating an AI that is at least slightly better than they are themselves, at at least the task of AI programming. If the author subscribes to the refutation that I quoted, then I think it follows that they also wouldn't believe this premise is possible to begin with.
Perhaps they'd also be brave enough to say what other things they don't think an AI will ever be better than humans at? Perhaps starting with the least impressive tasks, so we'll find out sooner if the predictions are correct?
> This ability of humans to build on one another’s work is precisely why I don’t believe that running a human-equivalent A.I. program for a hundred years in isolation is a good way to produce major breakthroughs.
As the saying goes, if I'm going to spend ten hours cutting down trees, I'd spend nine of them sharpening my axe. Just as a society of humans can develop tools to improve their capabilities, so can a single human, as the author's own example of Isaac Newton shows. If the AI is going to be locked in isolation for a hundred years to write software, why doesn't the author think it will spend some of that time on tools to improve its own productivity?