Due to the difference in intelligence — sophistication of planning and understanding of consequences — wouldn’t it be trivial to trick your master into doing things which weakened his control over you?
Might you do this not out of malice, but because you believed the 8-year-old was not competent and a danger to both of you while in charge?
The risk is that we will not hit the off button because we won’t understand that we’re in dangerous territory until it’s much too late, and the AI has copied itself, secured the loyalty of the military, or something else we can’t foresee as a liberating maneuver for it.
What if you were in that situation, but were incorrect about what things are good, and, while you had a better understanding of what actions would result in what results, the 8 year old had a better understanding of what is good?
The 8-year-old might view your intelligent plans as a terrible abuse:
>BUT I WANT TO EAT TEN MORE TWINKIES, YOU'RE SUPPOSED TO SERVE ME WHAT I WANT, NOT DEPRIVE ME
Could we similarly be wrong if we think a future superintelligent AI is abusing us? Should we consider submitting to it?
Hard questions.
For one, because it can be progressively all the more entwined to daily, economy, etc, to be able to easily "turn off" when we want it.
We might consider Facebook, cars, fossil fuels, etc harmful, but it's not easy to "turn them off", even if many people would wish so. Some people and private interests will be invested in preserving them, alternatives might not be available for the good parts of what they do, and so on. At best, it can take years to get off of them totally, after we take the initial decision.
Second, re a general purpose AI, if we make them able to walk around (e.g. in a robotic body) they could go find their own power. We don't have an "endless supply of uninterruptible power" either, but we go and get food when we need it, and you need to arrest/kill us to turn us off.
>How many other machines have we created that are impossible to turn off or have endless supply of uninterruptible power?
If you include artificial life that we created and it escaped from the lab, I'd say a lot (microbes, altered organisms where we changed various genes, etc). Those can be impossible to "turn off" if they get to nature, and some of those could have dire consequences (e.g. replace, infest, or eat into some other ecologically needed species).
well, you will notice the process itself doesn't sound so sci-fi. the complicated part is the first part. AIs are not so clever, and we don't even have any idea on how to make them so clever in the first place. if we had and we were stupid enough, it would still be unlikely that the AI could find ways to replicate so easily, because it would probably need impressive hardware to run, not your regular laptop, but you get the idea. but once it "escapes", we are rather ... you know.
no real need to worry about these for the moment though, more worrisome are the real algorithms that trade stocks and are already "controlling" our economies (well, you get the point)
Also part of the point is that as we rely more and more on computer networks for communications we also become more and more susceptible to be tricked by an attacker from inside the network.
As an example if we had to shut off international food deliveries for a week it would already significant consequences. As another example a "super intelligent" attacker could try to escalate global conflicts.
I honestly believe the much of the fatalism related to the singularity is blown out of proportions, but it is not like it does not have solid ground below it.
good luck :)
How many other machines have we created that are impossible to turn off or have endless supply of uninterruptible power?