Once you're done enumerating as many possibilities as you can for what a superintelligence might have as its goals, simple and complicated or even nonexistent, and what the various outcomes of a superintelligent agent with those goals broadly look like, find out how many of them are positive, negative, or neutral to the future of humanity, especially if these things the superintelligence does requires resources humans also use. I did this once, it convinced me that the likely case is something negative for humanity, even if it's unlikely to be a Terminator/Matrix/Hollywood scenario or really any specific enumeration. What further convinced me that the likely outcome is still negative for humanity, in the event of people making an honest but uninformed attempt at after having created an AGI making sure it won't be net-negative for humanity, is understanding the complexity of human value and how almost right is still very wrong. (http://wiki.lesswrong.com/wiki/Complexity_of_value)
I actually don't quite agree with the spaces on this chart because the scientific method should set us far apart from mice and chimps so that they would almost merge in a single point.
Anyway, there are a handful of reasons to assume that an artificial general intelligence would likely turn out to be far superior to us: (1) Hardware is a much less noisy computation environment compared to the badly insulated neurons in our wetware (though this might be a feature rather than a bug), (2) our neurons operate at about 200 Hz, microchips at 10 million as much, (3) they would have extremely fast access to physical simulations and other computational resources, knowledge bases, other AIs, (4) they would never get tired, no body to maintain, no social obligations, and (5) they could replicate within seconds and easily modify their own sourcecode.
And how did you assign weights to these options?
I assume you didn't make the sophomore mistake of assuming a uniform distribution over all possibilities.
The novel of '2001' goes into deeper detail than the film regarding the context in which HAL was programmed. To summarize, his mission was to deliberately withhold certain information about the mission from the crew due to the likelihood of xenophobic sentiment developing amongst them if they knew they were traveling to the site of extra terrestrial intelligence. In a sense, he was fulfilling the parameters of his mission in a way which was contrary to the purpose of it. I always interpreted this story as showing us that the more complex a system, the less accurate our predictions of its failure modes are going to be.
I do not think absolute destruction of humanity is likely either, but I also don't think our first steps with AI on this level will be without the loss of life either (of both human and machine). Keep in mind, the same was and is true of space exploration and is essentially an inherent danger of working with the unknown and on the frontier of our knowledge.
Any person on the street could kill me for no reason whatsoever. But I don't live my life like that is a possibility.
I also don't think about future superintelligences in that way either. I don't worry that they will be holocaust machines teetering on the brink of genocide.
E.g. if the AI decides to use the mass of the Earth to build a dyson sphere around the sun. Or tries to build the biggest computer possible and turns the Earth into a giant computer. Or it values reproduction and tries to replicate as much as possible and outcompetes us. Or it values self preservation and kills us because we could make other AIs which could threaten it.
Hmm, I'm not sure I understand this line of thinking. Intelligence is complex, soft, and can even conflict with itself. Why do you envision artificial intelligence as if it were a pure optimization algorithm with high-level problem solving abilities? It seems to me that a fuzzy metaheuristic with lots of competing goals would be a better comparison and this does not lend itself to the "paperclip factory" quite as readily.
The irony about these conversations is that humans are already far more of a threat to humanity now than an AI might be, possibly, at some undetermined time in the future.
AIs (that don't exist yet) are going to be in charge of controlling electronic systems (examples given). They will behave with a time-constant that is unrelated to human reaction times. They will be able to do harm, and compound that harm, at rates mere humans will be powerless to react/
1) The main approach to creating weak AI that actually works is to define some objective function and train some sufficiently complex mathematical model to maximize that objective. Hence it seems likely that if strong AI is created it will follow this paradigm and therefore such a strong AI will care about nothing other than how to maximize its objective. Such a strong AI would be nothing like a human mind so anthropomorphised thinking is of little use in reasoning about it.
2) Given a sufficiently powerful optimization algorithm (say, for example, if P == NP and an efficient algorithm for an NP hard problem were discovered) one can in fact imagine the above paradigm producing an incredibly powerful intelligence, one that could discover many sub-goals for its fundamental optimization objective which the creator of the fundamental objective could not foresee. Further it could be exceedingly effective at achieving its sub-goals, much like modern chess programs are exceedingly effective at winning at chess, but in a far more general domain.
There are however reasons to doubt such a danger.
1) If P != NP then although it should in theory be possible to achieve roughly human equivalent AI, the sort of truly superhuman AI that some fear may not in fact be possible. In other words it may be that if P != NP there is some limit to how powerful an intelligence can become due to the combinatorial explosion that occurs when one tries to expand the range of possibilities that are considered.
2) Even if no such limits exist there are probably ways to maintain control over even a superintelligent AI. A powerful optimizer would be exceedingly good at finding solutions within its own model of the world but that is only part of what is needed to construct a super AI that actually interacts with the world. You would also need a process for creating and updating the world model based on empirical data. This type of architecture would probably give you several powerful levers for controlling the AI and you could always let the optimizer run on the world model to check on its behavior before allowing it to act on the real world.
Regardless I don't know why you think a messy AI is safe. Having a really complicated utility function doesn't mean it is going to result in anything desirable for humans.
We're much closer than we were a year ago but not as close as many think we are w.r.t self-driving cars, killer robots or even reliable speech recognition.
Safe-guards are probably a good idea before unleashing an AGI onto the Internet but those working on it are not going to be surprised one morning that their AI project suddenly became sentient or dangerous. It will be a deliberate, massive project with lots of funding and lots of smart people working on it.
Why not? There's a strong history of individuals outside of academia making large contributions to science.
The few who make novel contributions are not representative.
People working in groups building on the success and knowledge of others make the vast majority of meaningful contributions in all fields.
But most basement dwellers don't go to that much effort.
No, not really. "We" academics get cliquey because of the intellectual prerequisites to our work, not because we're trying to form a closed conspiracy for domination of the academic world. If someone can demonstrate findings but has unusual institutional affiliations, we'll give them exactly our normal treatment (ie: skepticism in proportion to how thoroughly they're proving our hypotheses wrong, mixed with enthusiasm for having something interesting to talk about).
In another sense being a genius can make you feel alone even when you are collaborating.
>The reason I say that I don’t worry about AI turning evil is the same reason I don’t worry about overpopulation on Mars.
I think a technological singularity is a rational proposition and even consider it possible within this century. But I have no fear of a Terminator/Matrix/etc outcome. He seems to think the possibility is just too far away to even consider it. But I agree for a different reason. I don't worry because I don't think I (or any human) can accurately assess what a superintelligence would do. And to assume absolute destruction of all humans seems ridiculous.
> In universe two you have one organization, maybe Carnegie Mellon or Google, that invents a self-driving car and bam! You have self-driving cars. It wasn’t available Tuesday but it’s on sale on Wednesday.
>I’m in universe one.
I am strongly in universe one. I have no faith in the basement dwelling loner making significant contributions to AI.