Seems to me that some isolated artificially intelligent computer sitting on a deserted island somewhere isn't going to cause much trouble. Rather, it's the combination of a totally connected electronic world with an AI entity (and the notion that the AI entity would kinda sorta be a "native" of the interconnected electronic world) that could cause a lot of trouble.

Of course there are other reasons why having everything be "connected to the internet" (light switches, refrigerators, nuclear warheads) is probably not the wisest idea... would an AI with bad intentions really be any more harmful then a human being with bad intentions who had access to every interconnected thing?

It isn't AI specifically. Any intelligence capable of taking over or shutting down a wide range of systems, or even just a few critical ones, needs to be closely monitored if there is more than a small chance of it causing massive harm.

Now, if that intelligence also far exceeds our own in certain aspects but is not empathetic enough, and if it were to have a keen understanding of how to control humans, and if it were much more able to quickly control and adapt software and hardware, then it would become a much bigger risk.

But, you're right- in theory, there could be a natural intelligence that could be the same level of risk as an artificial one. Most of us, including myself, are just not aware of one.

> Any intelligence capable of taking over or shutting down a wide range of systems,

That's a thing I don't care for about most science-fiction. The AI entity can just magically take over every technology it's connected to, because magic. No concern about the computational complexity of breaking the code-signing certificates on the affected computers. It's like everything on the Internet of Things is hopelessly and completely insecure!

... okay now that I type that out, maybe that's more realistic than I gave it credit for after all ... :b

You can scale an AI's power more easily.

I'm not sure about that. A person can theoretically be scaled by adding another person to their cause, and there are a lot of people around. If the first brain in a box is simulated by a supercomputer that has only three others like it in existence, it's going to be much more difficult to scale it.

Of course, just adding another person doesn't necessarily make things go faster (mythical man-month and all that). But then again, just adding more hardware isn't going to necessarily make a brain in a box more powerful, either.

With AI you can optimize or change the underlying algorithms, or allocate more computing resources as necessary. You can't do that with humans—at least not in any literal sense, nor in any way that is remotely as efficient or effective.

The very nature of human intelligence is such that its architecture, scale and speed of computation are all bound within a relatively narrow range.

Humans don't scale. If you try to add more of them, you get things like governments and markets, i.e. lots of people who can't agree on shit. Not very efficient. Scaling a properly designed AI, on the other hand, could lead to something much more powerful - cognitive power of great many humans with a clear focus of an individual.

I'm not sure why you think humans don't scale. Do you think that the research done by the Manhattan Project was no more impressive than research conducted by a single individual? Or that a single individual is going to have as much success gaining political power as a political party?

Now as I said, "adding another person doesn't necessarily make things go faster." It can, but not always. But what evidence do we have that a human level AI would be able to scale easily? That seems to be a pretty huge assumption that gets glossed over.

>But what evidence do we have that a human level AI would be able to scale easily?

The fact it would probably be software, executing on a digital platform. These things typically lend themselves to optimization and scaling.

Seems to me that the chance we'll have just enough computing power to run human-level AI—without sufficient resources to go beyond that—is a small one. It is far more likely that we will either have too little, or more than ample computing power available.

What evidence do you have that human-level AI won't scale easily?

The best argument I can think of in your favor would be prohibitive computational complexity requirements, but the aforementioned dynamic still applies—it's unlikely that we'd have only just enough computing power to achieve human-level and not go beyond that.

We can look at efforts to stimulate the brain, which do not seem to scale easy. As I wrote in another comment, the K computer was able to simulate a seconds worth of brain activity. It doesn't seem to have scaled easily. I don't have experience with supercomputers, but I have a feeling that things are a bit more complicated than "just network it together with another supercomputer, and then everything will be twice as fast."

And that's just talking about a raw increase in computing power. We don't have any idea whether a human level intelligence program would even be able to put increased processing power to good use (and it could theoretically be detrimental; look at what happens to humans who have too many neuronal connections).

>We can look at efforts to stimulate the brain, which do not seem to scale easy.

Simulating the brain isn't necessarily the path to human-level AGI though. Simulating biology in digital form tends to be inefficient.

Modern machine learning scales easily, or at least relatively easy.

>(and it could theoretically be detrimental; look at what happens to humans who have too many neuronal connections)

Perhaps a constraint of our brain's cognitive architecture and not intrinsic to cognition itself. Still, interesting point.

Human minds in meat don't scale. But upload, and make a bunch of copies, introducing more-or-less random variation. Run in parallel, with suitable aggregation and selection. Got that from Quantum Thief by Hannu Rajaniemi.

You shouldn't consider government inefficiency a flaw. That's by design. If you have a government or a brain that makes decisions quickly, you might not have time to backpedal on seemingly good ideas that turn out bad.

I was thinking more in terms of cost. Adding a person to your cause requires a lot of money (the cost of them devoting their entire working time), and you only get so much extra benefit (efficiency losses due to communication and their own incentives). And the more effective the human, the more he would cost.

With an AI, you just dump that money directly into the brain by slapping on more compute power and storage. It'd be a direct increase in ability to its will. I'm not sure what exactly the raw costs would be (maybe it would still cost more than hiring a human), but at least it seems like it'd be a much more efficient use of each dollar.

The supercomputer that just simulated one second's worth of brain activity (K computer) cost $1.25 billion to design and build and $10 million a year to operate. Now, I don't have experience with supercomputers, but I doubt that you'd just be able to slap in some extra ram and get a big speed up.

It seems likely that the first simulated human level intelligences, at whatever point in the future they pop up, will at first be extremely costly and extremely slow. We don't really know if, say, connecting two supercomputers two simulate twice as many virtual neurons would make it more effective or if it would make it too unstable to do anything (how would a human brain with twice as many neurons react? It might just be mentally unstable and unproductive). But even a simple doubling experiment like that would probably take a long time to plan out and realize.

This analogy would be accurate if humans had brain-to-brain interfaces. AI's that collaborate will be able to do so much faster than humans with their slow politicking. But in before that, an AI doesn't even need to collaborate with other entities, as it can expand its computing resources at will.

The same thing that makes it easier for self replicating nano-bots to exist than self replicating macro-bots. Hand waving.