That's one of the best analogies we've got with regard to "deep learning" versus reality. People around here seem to think the AI apocalypse is 3-5 years away and are rushing to fund billion dollar "sentient rock" research.
Myself, I'm going to remain much more worried about the natural intelligences which have been hooked up to nuclear weapons systems.
It could be hooked up to the stock market, and it could make entirely rational decisions based on its objective (profitable trades) and these actions can result in imbalances leading to famine in certain regions, increased pollution, unsustainable depletion of natural resources etc.
We are already hooked up. The AIs are just amplifications of our own narrowly focused objectives.
The sort of long range damaging activities you mention are unlikely though, as algorithmic trading systems in general take their long term cues from humans.
If you're in the "2 year old with unlimited power" camp, nothing can save us and everything is futile and we should all just eat drink and be merry for tomorrow the AI kills us all.
If you're in the AI-as-enlighened-buddha camp, the godlike AI will either save us all — or — just leave us alone to solve our own problems (while potentially locking out future godlike-AI development so we don't do too much runaway damage (eschaton, etc)).
The powers of a GOD? What does that even mean? No AI is going to be able to go "Let there be light!" and make there be light. Heck, no AI is going to be able to go "I will hack into this camera and spy on you!" without either spending the requisite CPU-hours to crack the passwords or encryption protecting it, or analyzing all its attack surface for weaknesses like a hacker. Computational complexity is REAL, P does not and never will equal NP (we just don't know how to prove it yet), and there are real physical limits on the computing power that you can fit inside a given volume and its energy budget.
AT ITS BEST, an AI will have the same powers as a civilization of humans working together using computers the old-fashioned way, only faster.
Thinking about it, it probably comes from our general perceptions (based on conventional software development) that computers will either not do something at all, or they'll do it blindingly fast. And even people who work with computers often don't really grok the difference between multiplying two big matrices, and pondering the best way to approach an unsolved problem.
This alone is enough to reconsider the argument you are making.
While an AI wouldn't have the power to fundamentally change the universe or defy computational complexity -- what they could do would be near enough to godlike in comparison to humans that such a fact barely matters.
Quick example: In order to get the passwords this is what an AI can do.
1: build tiny robots. Something resembling a fruit fly.
2: robot fruit fly waits for appropriate person to use password.
3:see what password was used
4:mission accomplished
This is just one way. I'm sure the AI can figure out simpler ways to get the passwords. Tell me this isn't god like. It Is like arguing that we'll never go to the moon.
If evolution was able to invent intelligence so can we. And it will be a god. It is only a matter of time.
And I'm going to build it. (Evil Maniacal Laugh)
Yes, I think it is trivial. At least it will be in hindsight. The brain exists, it is proof that AI is possible just as birds were proof that flight was possible. We just need to discover the easiest way to implement it.
Nice read at the end of the article:
"[...] Naturally, this does not qualify as "seeing" in any human sense, and from a scientific perspective it certainly doesn't mean that we somehow solved computer vision at this point. Don't believe the hype; we are merely standing on the first step of a very tall ladder.
Some say that the hierarchical-modular decomposition of visual space learned by a convnet is analogous to what the human visual cortex does. It may or may not be true, but there is no strong evidence to believe so. Of course, one would expect the visual cortex to learn something similar, to the extent that this constitutes a "natural" decomposition of our visual world (in much the same way that the Fourier decomposition would be a "natural" decomposition of a periodic audio signal). But the exact nature of the filters and hierarchy, and the process through which they are learned, has most likely little in common with our puny convnets. The visual cortex is not convolutional to begin with, and while it is structured in layers, the layers are themselves structured into cortical columns whose exact purpose is still not well understood --a feature not found in our artificial networks (although Geoff Hinton is working on it). Besides, there is so much more to visual perception than the classification of static pictures --human perception is fundamentally sequential and active, not static and passive, and is tightly intricated with motor control (e.g. eye saccades).
Think about this next time your hear some VC or big-name CEO appear in the news to warn you against the existential threat posed by our recent advances in deep learning. Today we have better tools to map complex information spaces than we ever did before, which is awesome, but at the end of the day they are tools, not creatures, and none of what they do could reasonably qualify as "thinking". Drawing a smiley face on a rock doesn't make it "happy", even if your primate neocortex tells you so.
That said, visualizing what convnets learn is quite fascinating --who would have guessed that simple gradient descent with a reasonable loss function over a sufficiently large dataset would be enough to learn this beautiful hierarchical-modular network of patterns that manages to explain a complex visual space surprisingly well. Deep learning may not be intelligence is any real sense, but it's still working considerably better than anybody could have anticipated just a few years ago. Now, if only we understood why... ;-)"