If the AI doesn't have its own intention (perhaps having intention is a prerequisite for being truly intelligent), it's conceivable that we would program it with an intention or goal, such "find the most efficient way to produce as many paperclips as possible". If you assigned such a task to a human, it's fairly obvious that the task is subject to other things which are important to humans (for example, don't enslave or kill people to build paperclips, don't replace strip-mine all our farmland to find more paperclip material, etc.). But since an AI doesn't necessarily share, understand, or agree with all the implicit values we have, it's myopic view of the world could cause it to destroy us as a mere side-effect of trying to achieve it's singular goal.

yeah but that's bad design or terrorism, there is always another person to blame here. I think we are talking here about machines that take the blame themselves.

I'm not sure what the purpose is of focusing on blame. You don't need to assign moral culpability to a machine for its catastrophically poor design to be something to worry about.

It 'wants' what we programmed it to want.

It's not that the machine decided that it wanted to build paperclips. It's what we've told it to do.

The _means_ to build paperclips however is where the 'intelligence' comes into play. We've programmed the drive to build paperclicps, but we've left it to the AI to figure out how to do this. The article that you've read assumes that planning and subgoals are necessary for the AI to achieve its goal. Eg. That it needs to understand that in order to make paperclips it needs materials, and the further insight that to acquire more materials it needs to remove the other consumers of materials (us).

if (nearby() == human) {dontTouch(); abort() ; }

There, i solved that. The question is under what conditions would a machine remove that instruction and how could those conditions be reached?

How do we define nearby? Do we mean physical contact? Does the fact that the atoms of objects don't "actually" touch each other due to repulsive forces come into play?

Considering that an AI system would be software and wouldn't be constrained to a particular robot body, what does nearby even mean?

Is it physical proximity? Is indirectly touching a human with a ten-foot pole allowed? What about manipulating the global economic system such that the programmer who could shut you down loses his job?

We can't just separate the whole system from the world, because the AI was presumably built with a practical purpose beyond "sit in a box and don't touch humans." Social engineering is a danger; imagine having it send an email to a lowly programmer, pretending to be an important executive, requesting that they remove this one pesky line of code from the AI's programming because all this "proximity alert" stuff is interfering with its ability to actually accomplish what we invested time/money to get it to do.

Define nearby(). Because if the AI starts eating chunks of our planet we will get hurt as well.

INB4 Asimov's Three Laws of Robotics don't work. Actually, a lot of Asimov's writing is concerned with exploring their corner cases.

I have come to believe that corner cases quickly turn fractal.

Once you think you have solved it, changing the "zoom" will demonstrate that your solution has created X more cases.

It may be a side effect of the halting problem.

What's a human then?

Is a mentally retarded person a human? An embryo? A person who's lost his/her legs? A person who has education?

auto HumanObject = nearby();

if(Humanobject.IsPregnantWoman()) { PoisionEmbryoButNotHuman(); } else { abort(); }

Don't think of "wanting" as a soul-like action of a little person in the machine. "Wanting" is a shorthand for anything that is the result of a goal system and autonomy. In the simplest sense, giving a drone a map of its surroundings and giving it a location as the goal for its path-finding optimization algorithms, the drone now "wants" to go to that location.