The Skynet scenario seems farfetched because we're very far off from that kind of AI.

But on the chance that some day we do reach that level of advancement, even if it's 100 or 500 years, can't hurt to prepare, right? Better to waste time preparing unnecessarily than to face destruction from improper planning.

Yes, it can definitely hurt to prepare, if you're wasting resources.

Try proposing that we prepare for an alien invasion, and you'll be laughed out of the room.

The difference is we know we will eventually achieve AGI while we have absolutely zero reason to expect an alien invasion.

Not at all. We have good reason to believe we will eventually make contact with an alien species. How do you know we will eventually achieve AGI? We humans are evidence it's physically possible. Well we space-faring humans are evidence it's physically possible we'll make contact with ruthless space aliens.

Let's say many experts thought an alien invasion was quite likely to happen in the next 100 years (the way many experts feel about AI). At that point preparation starts sounding pretty sensible, no?

In my opinion the biggest danger is letting AI to sort our news and search engine results, social media feeds etc. There was research on Facebook how they can affect people's moods by using different weights for posts. What happens when intelligent bots start writing news, blogs, comments, tweets?

In essence, I mean the dangers of using AI for large scale propaganda through Internet services. The best tools of the most dangerous people and movements have always been manipulation and propaganda; what if a perfect AI does it? Could we even notice it?

Even when the AI is given a seemingly safe task, such as "optimize for clicks" in a news web site, something dangerous might happen in the long run if that's optimal for clicks.

Short-term, yes. What's the filter bubble already, if not the outcome of a super intelligent centralized AI silently and invisibly deciding what's best for us to see, molding our own artificial world dynamically based on our supposed preferences, excising away every possible serendipitous misalignment with our digital self as it is perceived by the machine?

I's easy to build uncensored search engines and news feeds to counter the bubbling effect. Much easier than building AI.

- Do you know you are an ad? south park ;)

I doubt it though. People got quickly desensitized towards advertising. Propaganda will follow.

People aren't desensitized to advertising, advertising adapted by using more subtle techniques designed to undermine our critical thinking by targeting us subconsciously. There's no reason propaganda dissemination couldn't adopt similar techniques. Think how scary a really smart AI that could subtly manipulate people's opinions would be, and how it could orchestrate populations to behave in who knows what manner, without them even knowing it's happening.

I like how your comment goes from no issues to alarmist.

You missed his point. People are the problem, not computers. Computers can help people hurt other people, just like guns help people hurt other people. But instead the hype is about the possibility of computers hurting people, on their own. Which, at least at this point, is about as alarming as the possibility that a gun will go off on its own.

And I think that you and the great-grandparent missed the point: It's not people, it's sentience that is dangerous.

Since the dawn of time that meant man or gods. Soon that list will need to include computers.

Until that "soon" arrives, there's a far greater chance that people will wipe out humanity. We have more suicidal terrorists today than ever in history.

Suicidal terrorists are hardly a threat to the existence of humanity. It takes quite a bit more than explosive vests to destroy all of mankind.

All of it, sure. But most of it might be easier than you think. Just one bomb in the wrong place can trigger a chain of catastrophic events. Don't forget that we have nukes in places like North Korea and Pakistan, and Iran is building one as fast as they can.

Excuse me, but I'd rather worry about those threats, than about a robot uprising.

"Stream of consciousness writing" or whatever it's called =)

I don't consider the two viewpoints expressed to be contraditory.

Powerful AI has already caused mayhem in the human world: http://www.wired.com/2008/09/six-year-old-st/

You can be malicious and destructive without traditional weapons... perhaps even more so with AI presiding over all our data. I mean we already have AI that answers our emails for us... won't be that long before its slipping things in that we don't notice.

This example of a human error has nothing to do with AI.

It's a small demonstration of the dangers of putting computer programs that do exactly as they are programmed in charge of making important decisions.

Let's say we program an AI and get one little detail wrong and things go to hell as a result. We can call that "human error" or "AI error" but either way it's a reason for caution.

I am actually somewhat concerned by this OpenAI project, and here's why. Let's say there's going to be some kind of "first mover advantage" where the first nation to build an AI that's sufficiently smart has the possibility to neuter the attempts being made by other nations. If there's a first mover advantage, we don't want a close arms race, because then each team will be incentivized to cut corners in order to be the first mover. Let's say international tensions happen to be high around this time and nations race to put anything in to production in order to be the first.

The issue with something like OpenAI is that increasing the common stock of public AI knowledge leaves arms race participants closer to one another, which means a more heated race.

And if there's no first mover advantage, that's basically the scenario where AI was never going to be an issue to begin with. So it makes sense to focus on preparing for the more dangerous possibility that there is a first mover advantage.

It seems like most people expect the emergence of the strong AI to be a sudden event, something similar to the creation of a nuclear bomb. However, it's far more likely that AI will undergo gradual development, becoming more and more capable, until it is similar in its cognitive and problem solving abilities to a human. It's likely that we won't even notice that exact point.

I'm not even sure governments are interested in developing AGI. They probably want good expert systems as advisers, and effective weapons for the military. None of those require true human level intelligence. Human rulers will want to stay in control. Building something that can take this control from them is not in their interests. There likely to be an arms race between world superpowers, but it will probably be limited to multiple narrow AI projects.

Of course, improving narrow AI can lead to AGI, but this won't be the goal, IMO. And it's not a certainty. You can probably build a computer that analyses current events, and predicts future ones really well, so the President can use its help to make decisions. It does not mean that this computer will become AGI. It does not mean it will become "self-aware". It does not need to have a personality to perform its intended function, so why would it develop one?

Finally, most people think that AGI, when it appears, will quickly become smarter than humans. This is not at all obvious, or even likely. We, humans, possess AGI, and we don't know how to make ourselves smarter. Even if we could change our brains instantaneously, we wouldn't know what to change! Such knowledge requires a lot of experiments, and those take time. So, sure, self-improvement is possible, but it won't be quick.

There's good reason to suspect that when AGI appears, it will quickly be developed to clearly super-human capabilities, just from the differences in capability between species that we are very closely related to.

Bostrom and others makes an argument that the difference in intelligence between a person with extremely low IQ and extremely high IQ could be relatively very small related to the possible differences in intelligence/capability of various (hypothetical or actual) sentient entities.

There's also the case of easy expansion in hardware or knowledge/learning resources once a software-based intelligent entity exists. E.g. if we're thinking of purely a speed difference in thinking, optimization by a significant factor could be possible purely by software optimization, and further still if specialized computing hardware is developed for the time-critical parts of the AI's processes. Ten PhDs working on a problem is clearly more formidable than one PhD working on a problem, even if they are all of equal intelligence.

How do you measure intelligence? If we take an IQ score as the measure, we will will see that many individuals with high recorded IQ, are not that remarkable when it comes to their activities. Usually they don't make huge advances in any fields, or become ultra rich.

We don't know if humans are 1000 times smarter than rats. Maybe we are 10 times smarter, or a 1000000 times. We don't know how much smarter Perelman or Obama is than a Joe Sixpack. We don't even know what "smarter" means. So talking about some hypothetical "sentient entities", and how "smarter" they can be compared to anything, is a bit premature, IMO.

>skynet/doomsday fear thing seems ridiculous

Maybe but better safe than sorry. Here's a scenario that OpenAI could protect against - super intelligent AI is first built by the US military, patented and backdoored by the NSA to protect us against terrorism. Then some one evil hacks them and turns them against us. Super intelligent AI being open source would reduce such risks.

> "predict where the this drone should strike next"

If there aren't military contractors or in house teams using machine learning for exactly that purpose I'll eat my hat. In fact they were probably doing it 10 years ago (for units, not drones), with tech we're only seeing the beginnings of now

Survey's of AI experts give a median prediction that we will have human level AI within 30 years. And a non-trivial probability of it happening in 10-20 years. They are almost unanimous in predicting it will happen within this century.

They were also unanimous it would happen last century?

What do we mean by "human level" anyway? Last time I got talking to an AI expert he said current research wouldn't lead to anything like a general intelligence, rather human level at certain things. Machines exceed human capacities already in many fields after all...

I don't think that many people predicted AI by 2000 with high certainty. And in any case, predictions should get more accurate as time goes on and we learn more.

"Human level" as in actually as intelligent as a human. An artificial brain just like a biological one, or at least with the same abilities.

I think we'll reach human level faster. We already solved to a high degree the problem of understanding the world. We can do perception for images, text, sound and video. In some perception tasks AI already surpasses humans.

We are also mining general knowledge about the world from the web, images and books. This knowledge is represented as feature vectors containing the meaning of the input images and text, or the so called thought vectors. We can use these to perform translation, sentiment analysis, image captioning, answer general knowledge questions and many more things.

On top of these perception systems there needs to be an agent system that receives thought vectors as input and responds with actions. These actions could be: reasoning, dialogue, controlling robots and many other things. It's in this part that we are still lagging. A recent result has been to be able to play dozens of Atari games to a very high score without any human instruction. We need more of that - agents learning to behave in the world.

I'd like to see more advanced chat bots and robots. I don't know why today robots are still so clumsy. When we solve the problem of movement, we'll see robots that could do almost any kind of work, from taking care of babies and the elderly to cooking, cleaning, teaching, driving (already there). We only need to solve walking and grasping objects, perception is already there, but unfortunately there's much less research going on in that field. I don't see yet any robot capable of moving as well as a human, but I am certain we will see this new age of capable robots in our lifetime.

On the other hand, we can start building intelligence by observing how humans reason. We extract thought vectors from human generated text and then map the sequences of thoughts, learning how they fit together to form reasoning. This has already been tried but it is in the early phases. We are very close to computers that can think well enough to be worthy dialogue companions for us.