Previous discussion:

https://news.ycombinator.com/item?id=7931674

https://news.ycombinator.com/item?id=7689579

What a terrible ending to the article. What Musk is warning against is a system that is capable of accomplishing it's own goals, and if those goals conflict with our own goals then there will be trouble - especially if the AI is clever enough. Whether or not we are living in a simulation is irrelevant - we are still agents with goals.

Maybe the author was trying to be silly, but, if that's the case I can see why he writes for CNet at not Comedy Central.

I think even before AI becomes intelligent enough to potentially destroy us. They will be intelligent enough to get used as smart weapons that are controlled by only a handfull of poeple that seek to controll the world. For Atomic bombs you still need a lot of people to keep them working and effective. AI Robots on the other hand could work, repair and advance themself.

This article adds to substance to Elon's cryptic tweet in favor of a cryptic reference to the Tron mythology. It seems to me though that the premise of strong AI warrants a degree of caution by its very definition. It would apparently be capable of all the things we are. But more likely it will be capable of all the things that we can program it to be capable of. the consequence of these actions remains underexamined, in my opinion.

Reminds me of Roko's basilisk:

http://rationalwiki.org/wiki/Roko's_basilisk

It's a bit annoying that you can't even talk about this stuff without the basilisk interfering. Maybe we're dooming ourselves by laughing at the basilisk incident instead of taking the potentially larger issue seriously.

Indeed, it's kind of a round about metaphor for John the Baptists 'repent ye for the kingdom of heaven is at hand'. Which also concisely demonstrates why many don't give the the singularity or related notions (even the possibility of strong ai) credence. It's a little to religious ie unknowable.