Well, yeah. But CERN creating a killer black hole would also be terrible. But we should think about what's probable, not just what's scary.
The fundamental principle of the paperclip argument is that the motivations of an AI will not necessarily align with our interests. An AI may do all sorts of things that seem nonsensical or morally repugnant to human beings if it does not share our moral intuitions.
If the intelligence of that AI significantly exceeds the range of human intelligence, we may be powerless to stop it or even to comprehend what it is doing. A rogue AI could become a catastrophically nasty Stuxnet, distributing itself across the billions of Turing machines we have networked together. Our only effective response may amount to "erase every data storage device in the world and start from scratch".
Related: http://blog.figuringshitout.com/nov-12th-day-30-no-evil-geni...
The idea in this thought experiment is not that the AI parses this sentence in the context of human culture (then it would likely comprehend that the actual intention is to maximize the economic success and eventually the human preferences of its creator). What is meant is that the objective is crudely implanted into the AI as an ultimate goal, in a similar way to how sustenance, pain avoidance and affiliation are very basic goals in our cognitive system. It is not entirely obvious that this is a stupid thing to do; hence the thought experiment. Will the AI suppress its urge once it comprehends human culture enough to understand the intentions of its creator? Will it rather successfully learn all the tricks to convert matter into paperclips before it considers studying human values? If the AI does not have a curiosity objective, it will likely not care about us very much, apart from the information that helps it optimizing its objective function.
Also, https://en.wikipedia.org/wiki/Intermediate_value_theorem
Also, what does the Intermediate value theorem have to do with it?
We are going to have a normal-intelligent AI before a superintelligent AI.
It'll see, but it won't care. See http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/
You might note that my argument is pure speculation without much basis in evidence. This is intentional, because it's all the arguments I've seen expounding on the AI apocalypse are equally speculative.
Intelligence and any consideration of its goals at all can be completely unrelated.
There is no reason for this to be the case. What something values is almost entirely independant of its intelligence.
At best, a human would be worthless to an AI who 100% only values paperclips (better as paperclips than humans), at worst seen as a threat to its ability to continue. (better to kill them off ASAP before try to stop me)
Thats the entire point of the paperclip maximizer thought experiment, to demonstrate that values, especially values humans care about, are not the same thing as inteligence.
Your entire ability to not "take things literally" is based off of a large amount of built in assumptions about whats important, your human nature and cultural experience. An AI would lack any of that, and if it was programmed only to value paperclips, even if was the most inteligent being in existence, thats all it would value, to the exclusion of everything else.
Consider autistic people with very high IQs, and due to their condition, very little understanding/concern about the emotions of others.
No one is doubting that machines fail to understand human values. The doubt is that such a machine could dominate its environment, or even work independently without careful management by humans.
The same thing could possibly (likely?) happen with an AI system. While it might be able to reason about the intent of it's design (what the code's supposed to do), it'll still pursue what "feels good", which is an artifact of it's actual design (what the code actually does, bugs and all...).
We're into some pretty deep stuff here. But the point is that no master creator has commanded us to reproduce infinitely and spare no expense. In fact our increasingly conscious control of our reproduction is reducing, not increasing, our risk of a catastrophic population explosion.
This idea, and many of Bostrom's other scenarios, boil down to "computers take everything literally", which is a rather cartoonish understanding of the concept of intelligence. It's possible that ASI will be programmed to have a hyperliteral view of all instructions, or will not be able to change its own utility function. But that seems like such a remote possibility, the exception rather than the rule. While superintelligent computers may pose many dangers, blindly executing instructions doesn't seem like the highest priority of them.