The machine does not hate you, nor does it loves you. But you are made of atoms, that it could use for something else.

---

If you create an AI with a genuine utility function. And that AI manage to be gazillion smarter than you are (think the difference between us and chimps, only much greater). Then it is likely the AI will (i) convince you to unleash it upon the internet, (ii) become even more powerful and totally unstoppable, then (iii) accomplish whatever it was programmed to accomplished.

If the goal is to keep humans safe and happy, it might imprison everyone in playgrounds, and drug or lobotomise any unhappy people, or restraint any suicidal people. Or, if you specified "happiness" with pictures of smiling people, it might resort to facial reconstruction, or it might just tile the solar system with molecular smileys, because that maximises the "make people smile" goal much better than actually making people happy. And while we're at it, happiness isn't the only thing we care about…

If the goal is to answer some difficult math question, it could tile the planet with computer, destroying the ecosystem and all humans in it, so it can compute the answer, and display it on a screen, for nobody to see it. Or maybe you programmed it to tile the universe with paperclips, because that goal is easy to debug. Oops.

---

As long as we're limiting ourselves to narrow AI (ordinary programs, really), we should be okay, and most of the consequences should be manageable. But as soon as we get to genuine AGI, we must worry about intelligence explosion and value alignment, so we can avoid creating a god that want something we don't.

Or we might end up with something a bit like the Culture - which is pretty much the best case scenario for humans co-existing with god-like AIs:

http://en.wikipedia.org/wiki/The_Culture

Vinge and Stross both hint at the worst cases - and they would be pretty bad, simply having using atoms re-purposed might not look that bad by comparison:

"There is life eternal within the eater of souls. Nobody is ever forgotten or allowed to rest in peace. They populate the simulation spaces of its mind, exploring all the possible alternative endings to their life."

The default outcomes are bad; Culture would be pretty great by comparison.

Worst-case scenarios are not likely for the same reasons best ones aren't - the default is AI indifference (they use you for atoms)

Since you mentioned worst, this is a must-read-classic :) http://hermiene.net/short-stories/i_have_no_mouth.html

>If the goal is to keep humans safe and happy, it might imprison everyone in playgrounds, and drug or lobotomise any unhappy people, or restraint any suicidal people. Or, if you specified "happiness" with pictures of smiling people, it might resort to facial reconstruction, or it might just tile the solar system with molecular smileys, because that maximises the "make people smile" goal much better than actually making people happy. And while we're at it, happiness isn't the only thing we care about…

I wish this would stop being repeated. The argument for a solar system tiling smiley face machine has been thoroughly refuted.

Would you be able to provide links to the arguments that refute it? Simply asserting that someone has disproved X isn't very helpful to the rest of us.

Seconded. I have seen a lot of smiley examples, and no serious refutation yet. A link would be very appreciated.

Thanks. I have read it all. I don't believe it.

First, the bulk of his argument hinges on the conflation of optimization power and value assessment. Those are easily separated. If you program an AI with a given utility function, and a goal of maximising that utility function, it won't modify the utility function in the course of increasing its optimization power. The utility function is supposed to be an invariant.

Now sure, you could try and tell the AI to align its own utility function to that of the human race… whatever that means. What is a human? How do you resolve conflicting values? It is all well and good to assume the presence of checking code, but he gave no hint about how those checks might actually work. If those are mere human-crafted checks, they are bound to be circumvented eventually, as the AI increases its optimization power beyond human level.

The AI doesn't care about human values. It only cares about maximising whatever goal it has been provided with, within the constraints of the checking code.

Second, I am wary of those swarm constraint relaxation intelligences. Since it is not even clear what utility function they might be maximising (if any), I wonder what would happen if they are given the power to self-modify in any significant way. Sure, this might be a lead worth following, but the safety of this design is even murkier.

Third, I'm not sure the guys at MIRI even believe in the doctrine of logical infallibility. They are well aware that perfect Bayesian reasoning requires super-exponential computational power, and is thus impossible to perform exactly. The AI is bound to approximate the Bayesian ideal, and cannot assume its reasoning is perfect —except in the few cases where it managed to produce an actual actual proof, like in 1+1=2, or simple probability calculations with simple prior information.

Overall, all I see is weak arguments grounded in assumptions I don't share. Worth a read, though.

It might be a little far-fetched, but there's a thought experiment called the "paperclip maximizer" that illustrates how even a well-designed, non-malicious AI could be an existential threat. See here: http://wiki.lesswrong.com/wiki/Paperclip_maximizer

i dont understand how that thing will develop intention. We perceive ourselves having it, but its really an accident that we do; if we didnt we wouldnt be here to observe it. Why would an artificial machine, no matter how smart , want to do anything?

If the AI doesn't have its own intention (perhaps having intention is a prerequisite for being truly intelligent), it's conceivable that we would program it with an intention or goal, such "find the most efficient way to produce as many paperclips as possible". If you assigned such a task to a human, it's fairly obvious that the task is subject to other things which are important to humans (for example, don't enslave or kill people to build paperclips, don't replace strip-mine all our farmland to find more paperclip material, etc.). But since an AI doesn't necessarily share, understand, or agree with all the implicit values we have, it's myopic view of the world could cause it to destroy us as a mere side-effect of trying to achieve it's singular goal.

yeah but that's bad design or terrorism, there is always another person to blame here. I think we are talking here about machines that take the blame themselves.

I'm not sure what the purpose is of focusing on blame. You don't need to assign moral culpability to a machine for its catastrophically poor design to be something to worry about.

It 'wants' what we programmed it to want.

It's not that the machine decided that it wanted to build paperclips. It's what we've told it to do.

The _means_ to build paperclips however is where the 'intelligence' comes into play. We've programmed the drive to build paperclicps, but we've left it to the AI to figure out how to do this. The article that you've read assumes that planning and subgoals are necessary for the AI to achieve its goal. Eg. That it needs to understand that in order to make paperclips it needs materials, and the further insight that to acquire more materials it needs to remove the other consumers of materials (us).

if (nearby() == human) {dontTouch(); abort() ; }

There, i solved that. The question is under what conditions would a machine remove that instruction and how could those conditions be reached?

How do we define nearby? Do we mean physical contact? Does the fact that the atoms of objects don't "actually" touch each other due to repulsive forces come into play?

Considering that an AI system would be software and wouldn't be constrained to a particular robot body, what does nearby even mean?

Is it physical proximity? Is indirectly touching a human with a ten-foot pole allowed? What about manipulating the global economic system such that the programmer who could shut you down loses his job?

We can't just separate the whole system from the world, because the AI was presumably built with a practical purpose beyond "sit in a box and don't touch humans." Social engineering is a danger; imagine having it send an email to a lowly programmer, pretending to be an important executive, requesting that they remove this one pesky line of code from the AI's programming because all this "proximity alert" stuff is interfering with its ability to actually accomplish what we invested time/money to get it to do.

Define nearby(). Because if the AI starts eating chunks of our planet we will get hurt as well.

INB4 Asimov's Three Laws of Robotics don't work. Actually, a lot of Asimov's writing is concerned with exploring their corner cases.

I have come to believe that corner cases quickly turn fractal.

Once you think you have solved it, changing the "zoom" will demonstrate that your solution has created X more cases.

It may be a side effect of the halting problem.

What's a human then?

Is a mentally retarded person a human? An embryo? A person who's lost his/her legs? A person who has education?

auto HumanObject = nearby();

if(Humanobject.IsPregnantWoman()) { PoisionEmbryoButNotHuman(); } else { abort(); }

Don't think of "wanting" as a soul-like action of a little person in the machine. "Wanting" is a shorthand for anything that is the result of a goal system and autonomy. In the simplest sense, giving a drone a map of its surroundings and giving it a location as the goal for its path-finding optimization algorithms, the drone now "wants" to go to that location.

>Now we are scared of something we haven't even created

I don't see the contradiction or strangeness in that. That's the whole idea behind "caution" -- to be wary of the potential outcome BEFORE it happens.

>How could the "interests" of a machine conflict with ours? I mean, they work with electricity, and we are hardly good sources for it

For one we are quite good consumers it. How about they get rid of us, so they have all the electricity production by themselves?

Second, as creators we'd want to boss them around (have them be "useful to us"). Why would they tolerate that?

Third, why risk us creating a potential competitor to them (another AI) rather than get rid of us?

Fourth, why not use us as cheap labor?

Fifth, as advanced AI, it could develop actual feelings, including sadism, greed, etc.

> Fifth, as advanced AI, it could develop actual feelings, including sadism, greed, etc.

I get that some AIs will have simulated reward signals, but letting them go haywire would just be bad design. We have nuclear heads, but we do not design them to explode at random. Now, our own reward mechanisms arise from the grander plan to keep organic life alive. Theirs have no underlying motive, and i don't think it goes the other way around, i.e you can't create a grander plan starting from the rewards. One would need a mathematical proof for the latter.

I think there's some over-anthropomorphization of the AI here, in terms of it picking up human qualities as if they were Things outside our brains that we possess.

But I feel like this particular argument is (please correct me if I'm wrong) saying something like "Well of course we wouldn't let them go crazy! That'd be ridiculous!" I agree -- a sensible solution would be to have good design and introduce safeguards. That's the point of people like Bostrom when they're advocating for this, because before they started writing about this, it seems like a lot of people weren't even thinking it could be a problem. How many systems will be built without good design, because they were made to solve a short-term problem. Will the finance bots doing transactions at the microsecond scale in order to maximize the assets of some particular company have been built to not let them go haywire?

>I think there's some over-anthropomorphization of the AI here, in terms of it picking up human qualities as if they were Things outside our brains that we possess.

I don't see any issue with anthropomorphization with regard to intelligent being (at any level).

To have an issue with that would be to assume that humans are more special than other life forms (and AI would be like a life form in this sense).

Animals can have empathy, anger, anxiety and other traits we associate with humans (and why wouldn't they? It's not like our brains are unique, they're just "more"), so AI could too.

>I get that some AIs will have simulated reward signals, but letting them go haywire would just be bad design.

The problem with this is that for real AI, you don't have control over what they feel. It's not like some program of the sixties or seventies AI, were stuff was "simulated", it can be modelling a real neural network that works like the brain, and into which you don't have much control over what will develop.

That is, the kind of AI people are afraid of is not "simulated", it's actually thinking for itself.

Because it's easy to construct angels and demons in one's mind, about anything. It polarizes the mind into fear and desire, forcing forcing a person to continuously oscillate between extremes. It makes it seem like there are always only two choices, when in reality, there are many, if not infinite.

Also, there are always going to be some people who understand the machine better than the machine understands itself. Even if it means taking those gigantic tables of probabilities and abstracting over them to form some kind of representational language that can be used to predict complex non-deterministic machine behavior (similar to how we already try to predict non-deterministic behavior, or not) in complex systems.

As a corollary, I would argue that machines have already modified humans. They affect the way we think and reason. They are an integral part of our life and the machine abstraction can be projected onto that which it does not describe completely (in which, the technology that exists now can not construct the technology we envision). We program machines, machines program us. This seems obvious.

If we managed to create an AI that gets out of control, we would pose an existential threat to it due to our ability to create another even more powerful and uncontrollable AI.

That's a good point. But the AI could do that too, and maybe better.

I would expect the first AI to improve itself (or produce a new AI) in a controlled and planned manner (from the perspective of that AI, not the humans). Intelligence explosion would look like an instant revolution to humans, but from the perspective of the AI it would be a lengthy evolution process.

The second AI created by humans is unlikely to have same goals or to be predictable by the first AI (or whatever evolves from it), thus I guess it's something that the first AI would want to prevent, either by eliminating humans or by fully controlling them (for their own good).

It's extremely important that the first AI we create is well thought out as we won't have the second chance to fix things up, once it's here it will have god-like powers over us.