Read Superintelligence by Bostrom to help see where they're coming from. It doesn't think AI will be evil either (unless perhaps those who develop it and determine its goals are...). But AI could run out of our control, or have us fall victim of unintended consequences.

A lot of people didn't understand this about 2001: Hal was actually doing the logical thing, he was doing his job. The human was unreliable and was indeed endangering the mission. Hal was a tool of mankind, just like the bone that the ape uses to kill in the first scene.

In the same way when we talk about the dangers of AI we tend to personify them and expect them to be either intelligently nefarious (skynet, matrix) or out of control (bug, malfunction).

What we don't imagine is that we might end up being on the wrong end of a perfectly rational decision, or that we don't realize the consequences of the process that we set them to do.

That is not a reasonable argument. They have demonstrated nothing but a fear and postulation and we cannot accept authority as evidence. Thus is the nature of science.

I never said we should defer to their 'authority' either. But it is a conversation worth having, and the sooner the better.

If someone more knowledgeable than me on a subject fears it, well I think it's worth at least listening to him. Not dismissing his concerns arrogantly.

And when those person are one of the greatest scientist and one of the greatest entrepreneur of our time, both with some kind of expertise in AI, well...

Stephen Hawking partakes in research, with Intel, mostly on speech recognition, to help him being able to better express himself.

Elon Musk funds several startups in the field of advanced AI.

And please stop with your arogant (and erronous) definition of what science is and science is not.

Higgs theorized the Higg's boson existence in the 60s, that could not be tested until 50 years later, are you saying this was not science? Einstein's general relativity predicted the existence of black holes in 1916, we still don't have definit, hard proof of their existence, is it not science? The Big Bang theorie has never been proven (data from last year BICEP2 experiment are still being dissected), is it not science?

Should we not talk about all this subjects because they are not proven?

What you are referring to, is the scientific method, which is the way we use to decide if something is sufficiently proven to be deemed as true or not, not the definition of science itself.

But how can you have something testable about something that does not exist yet? You would have to have evidence that an actual, intelligent program does something hostile, which is obviously impossible at the time since AGI has not been invented yet.

However, we do have evidence of naturally occuring hostile intelligence. If an AGI is vastly superior in some regards (for example through much tighter integration in knowledge and control systems), it could in fact pose a danger. The article even clearly states a hypothesis: "Humans, who are limited by slow biological evolution, couldn't compete and would be superseded."

We can’t really assign a probability of occurrence to scenarios like these, thus we have to include it in the list of events that could potentially wipe us out and which we may be able to prevent. Since people are actively working on AGI, it should probably be somewhere at the top where nuclear warfare, global warming and comet impacts are listed.

So you can choose to wait and see, and maybe it will be too late or to hard to act when you have sufficient proof.

Or you can choose to start thinking about it now, and see what we can do to to set up safeguards. And perhaps those safeguards will be useless, perhaps not. That's called prevention.

You can do research and testing without fear mongering.

While I agree that worrying about it would be silly, the only thing I know about strong AI is that I don't know what the implications of having strong AI would be.

It’s not clear how sophisticated a neural network needs to be in order to yield AGI. It could be very simple (then you would be wrong with high probability), or it could extremely complex (then you would be right with high probability).