What we are skeptical of is that we need just a little bit of development and strong AI will take-off just like that without any regard to physical world constraints.
Really? Brains need food (carbohydrate ->-> atp). Last I checked, computers need energy.
Also, the upper bound on how much energy is available to us should we really need, from the nearest power source, is some 6-9 orders of magnitude greater than the amount we currently take from it as a whole planet (most of it being wasted). Food growth, on the other hand, is much more limited by many other factors such as ecosystem impact, limited land area, failing crops, speed of production of seeds, and all sorts of pesky logistical issues...
If our hypothetical AI saturates insolation output and minimises waste (most of our 5% usage is utter waste from a computational point of view. We mostly spend our energy to produce poop and move physical stuff around pointlessly), it will already have a lot more energy to spend on computation than we do today, and then if it really needs more there's millions of times more energy available just one AU away, easily harvested if it's already figured out how to do so efficiently on Earth.
So my point is, there may be a Malthusian limit of sorts, but by the time we reach that, the bottom part of the S curve is so far behind that we can't even remember it anymore. It's as if the classical Malthusian limit was that Earth can "only" feed 10 million billion people.
First of all, this is not an attack on the singularity idea, but on the "strong AI" idea.
With respect to strong AI, Nick is very close to the problem, and that's the issue. Like any good engineer/scientist, he sees problems everywhere. Yes, there are many problems before we get to strong AI. That's not news. There were many problems to resolve before people could pay a small sum of money and let a giant metal tube take them through the air to a destination halfway round the world without killing them - but those problems got resolved, one by one. Many problems do not amount to an impossibility, only a damn hard problem (which we knew strong AI was anyway).
As for his other assertion, that the concept of the singularity is a fantasy, Nick's main argument is that the singularity will only last "for a time", and that it will turn into a classic S-curve. He waves Feynman's name around as supporting evidence, but does not address the fact that intelligence (and artificial intelligence in particular) is not subject to the Malthusian laws which have caused other phenomena to follow S-curves. Yes, we only have access to so many particles, but the whole point of exponential AI is figuring out better ways to use the same number of particles. There may be a theoretical limit to how efficiently we can use those particles, but even so there are a lot of particles, and if we can manufacture even just human-equivalent computing matter in factories, that's already enough to achieve a singularity.
So, in summary:
- Yes, there is a problem with Singularity stuff being quasi-religious (I've been to a Singularity event and agree - though the proportion of kooks was relatively low, there were certainly a few, some of them on the podium speaking)
- No, this article is not a convincing argument against either strong AI or the potential of a singularity-like event to occur.