I think financial algorithms are probably the best non-toy model of dangerous AI, simply because it's easy to see the flow from optimization to real-world consequences.
The problems with end-of-the-world AGI scenarios are practical, not theoretical, and I think a lot of people get hung up on arguments such as, "Of course we wouldn't allow a paperclip-manufacturing AI the ability to deconstruct human beings."
In a larger sense, I think Bostrom can restated (in part!) as the trivial observation that "optimization is insensitive to hidden variables," and it describes any kind of complex system, not AI and AGI alone. For example, the failure to capture environmental remediation costs in market pricing structures means that we improperly discount future costs for things such as global warming or lead exposure. In health care, the implementation of the Prospective Payment System led to hospitals dumping patients back on the street "quicker and sicker" to maximize margins on care. And thousands of financial systems focused on microsecond returns don't pay attention to, or even have any way of understanding, how their coordinated activities can be macroeconomically destabilizing. Any time you allow outcome optimization+general reasoning+real-world outputs, you have the opportunity for a killer paperclip scenario.
They are not self-aware, or superintelligent. In addition to that we monitor them and can react if the decide to crash the financial system just to make few more bucks. Paper Clip machine will be a robot, not AI.
My problem with this whole example given by Bostrom is that if the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless.
I fully understand that this paper clip example is just oversimplification, but I would expect more from people spending entire lives thinking about that stuff.
It's nothing but a fear-porn designed to get more funding.
This is an interesting point.
I'm not convinced that it could decide or figure out that the operation is pointless (for reasons I'll explain below), and I feel like it is perhaps taking the paperclip example to an unnatural extreme.
Really, the crux of the matter is that any intelligence agent at human level would have the intelligence and problem solving skills we do without the moral and social constraints that we do. Any problem it decides to solve or any task it's given to complete will be performed with the same ruthless efficiency you'd expect a machine to have. This is why it might not be able to figure out that the operation is 'pointless' or 'wrong'. If you give it an objective but don't provide first order or axiomatic values for human life or the non-substitutability of people, then you'll get behaviors which you would consider evil.
Consider the following:
"Person arm stuck in assembly line. Halt assembly line for 1hr, lower-bound for total cost: $1,200,000. Continue assembly, upper bound for injury settlement cost: $1,000,000. Decision: continue assembly."
There's no implicit consideration of human worth that people have because human morality is relative. In general, we have a similar set of base parameters instilled by evolution (cute things and things like us are good -- don't hurt or kill them), but we can't prove from any mathematical axioms that these are 'right'.
This is why people get scared -- if you program a computer to maximize the profits of a company and to do so at any cost, a human-level AI will spend an eternity doing just that, even if it means the complete subjugation of humanity. (Since we can't eliminate humanity otherwise currency ceases to exist and economic collapse will hurt the bottom line.)
I would wager (but can't prove) that a study of abused people, child soldiers, and sociopaths would yield interesting parallel psychological profiles. These are people (human level) with a set of axioms which are distorted by artificial means. There's still the built-in problem of the human component adding "Don't hurt the cute things," in most cases but nonetheless.
I have written algorithms for autonomous programs like this. Pretty much advanced IFTTT, automatically analyzing the situation and reacting.
The program is not aware of any values other than those it is programmed. Obviously, it will not regard or reflect upon its actions and the effects thereof outside of optimizing the profit from trading.
Any definition of "general intelligence" would have to include the ability to revise and reconsider ones own goals and purpose. An AGI that is forever bound to maximize the paper clip output at any cost, does not, in my view, meet that definition.
But I grant you that there could be "higher" goals that are incompatible with the existence of humanity. Making paper clips just is not it.
There is a whole world of more complex failure modes. "Make everyone happy" leads to wireheading or heroin addiction for the population, unless you grant a more compelling Theory of Happiness than any two human beings can agree on. "Make everyone food to eat", "Cure my PTSD", "Set up a colony on Mars" - unless you define all these things in a way that is specific not just to humans, but to modern culture, you leave open the possibility of undesired results. Mythology is thick with gods and djinn who grant wishes but turn out to be tricksters.
It's not that it "wouldn't necessarily have to share our values" - it's that there's no way in hell these things would share our values; They actually can't share our values until we solve "our values" as a complete mathematical proof, and then figure out a way to code that. In a resilient way, without any chance of the neuroses or obsessions or self-deceptions or delusions or corruptions of perspective available to humans.
Solving philosophy is a precursor to a Friendly AGI.
Good luck bug-fixing.
General artificial intelligence is an antropomorphic concept in the first place. It includes inefficiencies. It includes self reflection. It includes uncertainty and disagreement about what is and isn't efficient and about what is and isn't right.
Does that guarantee that any AGI would be reasonable or even morally acceptable in our sense? No. But if it is intelligent in our sense we know two things:
This entity can be reasoned with, and having grown up in our midst, it will understand us. It would also be very surprising if it shared none of our values. After all it has evolved among us and any intelligent being is going to pick up some of the values of the society it is part of.
Secondly, intelligence includes creative leaps. That means, no two AGIs would share the exact same opinion, just as no two humans do. So we may be able to convince a majority of AGIs to be kind to us, and any efficiency of an AGI society wouldn't be as ruthlessly efficient as if there were just one single all powerful AGI.
So, all I'm saying is, let's not strip the term general intelligence of all its non-mechanistic, self-referential, social traits. An AGI could one day anihilate us but it would probably not be a side-effect of some fairly unimportant industrial process like making paper clips.
It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little homonculus in a box. Rather than thinking of it as "an intelligence," maybe think of it as "a system that is incredibly effective at carrying out its goals in a general manner." Making paperclips doesn't require super-intelligence, but that doesn't have anything to do with the premise, because intelligence can be turned toward any goal. Think about modern financial trading algorithms -- they're probably built to maximize the money that flows in, and there's nothing that says "I know that this money will be used for X." Think of most AGI like psychopaths/sociopaths -- certain actions that don't play nice with others aren't because they're dumb but because there's intelligence without things like empathy to go along with it. It would be easier to build an AGI focused on the goal of making more paperclips than it would be to build an AGI focused on the goal of paperclips but also aware of all the fuzzy situations where a human wouldn't want that exact goal and why it should even care.