I'll admit this is purely hypothetical, but given that we know many people act on stuff they read on the Internet in an uncritical fashion - we cannot dismiss AI because it "doesn't have access to air gapped systems". A human has access to that system - a superintelligent AI could convince that human to "push the buttons".

I don't actually believe this will happen, but I'm not 100% confident it wouldn't. I don't know what a superintelligence would look like!

Then your problem is humans, not AIs. And "pushing a button" is a severe understatement of what an AI would need to have a human do to cause the implied level of damage.

I have read some stories where various airgapped systems were "compromised" in some way (microphone picking up CPU harmonics,etc). See here for example: https://thestack.com/security/2016/06/24/even-speakerless-ai... Obviously those are fringe cases and not practically exploitable outside of a specific environment today.

What I think the danger with a super intelligent AI is being able to correlate massive amounts of data to see patterns we miss or things to try, and never being bored. Look at automated exploits today vs the old "hacking by hand". Brute force attempts not just on known patterns, but being able to create new tests and analysis.

The power of an AGI goes far beyond intelligence. It could generate millions of its own instances (possibly with variations) and hack on thousands of systems at the same time, actively adapting each one as needed. It could perform social engineering on a large number of people simultaneously and communicate in ultra-high bandwidth and protocols that most humans involved cannot follow.

A likely potent form of gaining power may involve assuming the identities of people one knows, via text messaging or email, and use that to gain access to important information. The aggregation and superhuman-level analysis of private information thousands of times faster than humans would bring enormous financial and social power to the AGI. Given the sad state of security in most systems, and the relative ignorance of information security issues among many political and business leaders, an AGI will not have much problem with this sample plan off the top of my head.

And there are many other more clever plans an AGI could come up with that we have not yet considered...

Suppose AI 1 uses that approach but Gmail has AI 2 acting as a spam filter. Suddenly phishing becomes much more difficult.

Further, spinning off other copies requires hardware to run them. If MS has an AI 3 go over windows source code it may be much harder to hack desktops.

Without authorial fiat a world of real world AI's may be much more stable and boring than you might think.

Defense is generally much harder than offense, in most systems. The attacker needs to find only one loophole. The defender needs to protect all of them, including the fallible humans in the loop. This is one reason even large corporations often fail against a small group of hackers.

And in the real world, not everyone exclusively uses services and systems from Google and Microsoft, presumably two of the most competent technology providers. Plenty of people also use ad hoc services set up by internal teams.

VISA, MasterCard, the US Federal Reserve, and IRS etc are a vast targets. However, while people regularly fake CC, Visa's internal systems seem safe. Suggesting defense is actually easier than offence when both sides are competent.

> More importantly, flawless security trumps hacking. If there is nothing to exploit there is no way in, which means a lone AI is much more powerful than lot's of AI's with different goals.

No security is flawless in the face of a sufficiently determined attacker. No system has nothing to exploit.

Airgapped systems have been attacked in the past. Stuxnet is a good example. No magic required.

Just goes to show that flawless hacking trumps security.

There's a subtle argument here philosophically that I think is crucially tied to AI: determinism.

If the world is deterministic, a sufficiently powered and advanced AI with enough data could actually predict the weather, not just for two weeks, but for years. This also enables many other things viewed as magic.

This, of course, is incredibly far off. And I don't see a flaw in your point on flawless security. Though, I doubt we have many, if any, truly flawless security systems in existence. We just don't know their flaws, making them good enough for now. The world could get really interesting once AI's start programming and writing app security to protect itself from malicious AI's.

Even with a totally deterministic model, you can't predict weather very far out because it's chaotic. In other words, you'd need to measure the parameters with arbitrarily high accuracy, which even an advanced AI can't do. To oversimplify, if you need one more bit of accuracy in the measurements for each hour you forecast, it's impossible to get enough accuracy to forecast a week out.

For a detailed discussion of chaos and weather prediction, see: http://www.ecmwf.int/sites/default/files/Chaos%20and%20weath... The "trick" for dealing with chaos is to do a bunch of predictions with perturbed values. If they all end up the same, then the forecast is probably okay. If they vary widely, then you know not to assign much accuracy to the forecast.

I agree that "determinism" has much to do with the questions here, but to immediately pose a question about "the world" being deterministic might be putting the carriage before the horse.

I think the the other, perhaps more prescient point here, is about whether or not that kind of exponentially-learning AGI being discussed is even possible given that the substrate such things would be embedded in (hardware, software) which are [or appear to be?] deterministic. One assumes that that kind of learning implies learning about or aquiring capabilites which have not already been determined or specified by a human. At some level of abstraction, humans _must_ determine that system. It is possible to concieve of a machine that is non-deterministic in that way?

At what point does such a system actually leap beyond what has been specified by us? Does this imply a machine that not only rewrites itsel in softwate, but also remanufactures itself in hardware?