In my opinion, the Middle Age alchemists who made gunpowder and first primitive bombs, didn't need to establish research programs to worry about advances in bomb-making leading to the threat of Mutually Assured Nuclear Destruction.
If they had tried (maybe they did?), maybe they would have come up with ideas like very heavy regulation of the trade of saltpeter, so that no one has enough to make a verybigbomb.
I agree that everything you described will happen in the future. But in my opinion we are at a Middle Age alchemist's level in AI research (no offense to AI researchers) and we can safely wait a 100 years, and let those people worry about the existential threat. They will be in a much better position to do so appropriately, because they will know much more than we do. And they will not be late, either.
It looks like we've got ways to go there though, current programs don't seem to even begin to act anything like an adult human. So if the problem was to engineer an out of the box adult human level AI, we might again assume that there's obviously decades of work left to do before anything interesting can be developed. The problem now is that that's not how humans work. Humans start out as babies and learn their way up to adult cleverness. I can tell that an AI is nowhere near having an adult human intelligence out of the box, but I'm far less sure how to tell that an AI is nowhere near being able to start a learning process that makes it develop from something resembling a useless human baby towards something resembling an adult human in capabilities.
[1] http://www.nickbostrom.com/superintelligence.html [2] https://plus.google.com/+JamesPearn/posts/gTFgij36o6u
While that argument would be sufficient to justify greatly increased attention to AI safety, I don't think it's the one that's being made here. A good overview of the argument by the Machine Intelligence Research Institute is at http://intelligence.org/files/IE-EI.pdf . They don't think the probability is small.
I wonder is this really at the root of a lot of people's failure of imagination. Maybe it's Hoftstadter and Dennett people should be reading rather than the technical AI detail.
In my view, qualia refers to a phenomenon, but tries to deny that it is a phenomenon by arguing that it is something independent from the physics of perception. I believe that if anyone ever tried to devise a machine to represent the boundary between physics and qualia, such as an artificial intelligence or brain, the hardware of that AI would provide a means to decoding subjective experience into objective terms.
Qualia would cease to be a useful term, and we would instead rely on a more objective look of individual perception, all while losing no advantages for abandoning "qualia" to the word graveyard.
The basis for my saying what I said was that the existence of qualia seems at least as self-evident as the existence of anything else, particularly if you engage in a practice like zazen, as I do. (I would guess that the people who deny qualia and do that kind of meditation are probably quite few in number.) But that doesn't mean some extremely logical and compelling argument that I can't imagine now couldn't possibly make me see things in some entirely new, transformative light.
If anyone would be discussing that Dennet overview with me after I've read it, please say so...
Consciousness is just a feature of that algorithm, in which it just becomes aware of its own existence and then ponder its implications.
OP is probably referring to the Hard Problem of Consciousness[1].
I think it's a real barrier. It's fundamentally illogical trying to map formal constructs onto experiential domains. But that doesn't mean building a sentient being is impossible. Our existence is proof that it's possible. It does mean the best model our brains can muster for sentients is a blackbox system. Some sort of state machine or statistical model built up from correlations.
[1] http://en.wikipedia.org/wiki/Hard_problem_of_consciousness
Even fairly simple quantum systems (which are "purely physical phenomena") cannot be emulated by any classical computer in any meaningful sense, since the computational complexity of integrating the dynamical equations is exponential. Even if we could recruit all the atoms in the known Universe, we still couldn't build a classical computer capable of emulating many simple quantum systems.
In the past 233 years we have evolved from the first steam machine to the iPhone 5S. I would say we have a pretty good track record at overcoming miniaturization problems.
If you open any Neuroscience textbook, there will be no mentions of quantum phenomena as related to consciousness.
Lifting constraints fundamentally changes some problems. Imagine the ability to simulate a population of a few thousand brains, with each generation run of a few minutes. Artificial selection is then enough to create an intelligent simulated brain.
By then, you may throw all current AI algorithms in the trash bin. Unfathomable to any self respecting AI expert today.
I'm sure we'll be able to develop computers that are better than us at solving increasingly general problems, but that is nothing like having a human-like brain, only much smarter and able to improve itself. It will still be a problem-solving machine.
I for one believe that it's far more likely humanity will develop a symbiosis with it's silicon counterpart that eventually will run so deep that it will become almost impossible to say whether we run the machine or the machine runs us.
The problem is that, even though I'm fairly sure my version is more realistic, in practice it's still just one untestable hypothesis amongst many.
"Any sufficiently advanced technology is indistinguishable from magic."
--- http://en.wikipedia.org/wiki/Clarke's_three_laws
The brain might as well be magic, I don't think we have enough understanding of it to know if it's truly possible to make a computer think & learn _exactly_ like a human.
Also, I can't help but be reminded of the TNG episode related to this topic. http://en.memory-alpha.org/wiki/The_Measure_Of_A_Man_(episod...
This is the only phrase I disagree with in your very thoughtful comment.
When was 'the cautious route' ever a scientific principle?
Either way, has it ever been applied in a meaningful way?
It is in constant use, although there are probably places in the world where it is given more attention than others. The United States is absolutely not the world's bastion of the careful and incremental application of new knowledge, this may be why it is unfamiliar to you.
A prominent historical example would be the reluctance to use the first nuclear bombs before establishing a consensus on whether such an explosion could ignite the atmosphere. Contemporary examples would be e.g. the care taken when using a new, promising drug prior to rigorous animal and human testing or the reluctance to use new useful chemicals and materials (e.g. nanoparticles) everywhere due to the possibility of long-term health effects like cancer risk.
First, for the most part, we expect that AI trained to do X is going to do X. Insofar that doing X is an AI's "survival criterion", you're not going to get rebellious AI populations for much the same reason you don't see species that refuse to reproduce. There's some safety in numbers, too: it's rather unlikely that all the machines we make would fail in the same systemic fashion. Furthermore, most realistic training methods are incremental, which means if there's trouble we'll see it coming. It won't just fail in the worst possible fashion, out of nowhere, without any kind of foreshadowing.
Second, AI will not evolve in a vacuum. It's unlikely we will jump from human-level intelligence directly to superhuman AI. By the time that happens, most systems will already be AI-controlled, so being a lot smarter than humans won't suffice. It will need to deal with hordes of loyal AI protecting us (or persons of interest).
Imagine the human-level ai had resources to run hundreds of variants of itself. And imagine that it had no compunction about killing copies of itself that underperformed. How long would it take to get improvement of 10-100x under that scenario? Once it has a full understanding of what works, the upgrades are instant from there on out.
Why would it do this? Because any goal you give it would be optimized by being smarter. I would argue there is no island of stability at human-level intelligence.
> Imagine the human-level ai had resources to run hundreds of variants of itself.
The first human-level AI will probably be hosted on cutting-edge hardware that costs billions of dollars. So no, it won't have the resources to do this. Even if it wasn't this expensive, it's not exactly sitting around a vat of free computronium that it can do anything with. If my AI takes up 10% of my computing resources, why am I going to give it the remaining 90%? How is it going to hack or pay for servers to simulate itself on? Is the first human-level AI a computer whiz, just because it runs on one? But let's assume it can and will do what you describe.
> And imagine that it had no compunction about killing copies of itself that underperformed. How long would it take to get improvement of 10-100x under that scenario?
If the AI was trained using a variant of natural selection to begin with, it won't improve itself on its own terms any faster than we improve it or its competitors on our own terms. If the AI wasn't trained that way, then probably never, because that's not how these things work.
> Once it has a full understanding of what works, the upgrades are instant from there on out.
Your scenario is still based on several dubious assumptions. If the AI is a neural network modelled after human brains, its "source code" would be a gigantic data structure constantly updated by simple feedback processes. It is dubious the AI will have true read access to itself, in part because there's no need to give it access, and in part because it's hardware overhead to probe a circuit comprehensively (it would be easy to read the states of all neurons on conventional hardware, but all of conventional hardware is overhead if you're only interested in running a particular type of circuit). Even if it could read itself, a "full understanding" of any structure requires a structure orders of magnitude larger. Where is it going to find these resources?
Long story short, a human-level AI probably won't be able to understand its own brain any better than any human can. Now, we will certainly have statistics and probes all over the AI to help us figure out what's going on. But unless we see fit to give access to the probing network to the AI (why?), the AI will not have that information.
> Why would it do this? Because any goal you give it would be optimized by being smarter.
Unfortunately, any resources it expends in becoming smarter are resources it does not spend optimizing our goals. If we run hundreds of variants of AI and kill those that underperform, which ones do you think will prevail? The smart AI that tries to become smarter? Or the just-as-smart AI that focuses on the task completely and knocks the ball out of the park? The AI won't optimize itself unless it has the time and resources to do it without an overall productivity loss and before we find a better AI. For human-level AI, that's a tall order. A very tall order.
No one is talking about magic. The human brain is not magic, neither is that of chimpanzees, rats, dolphins or gorillas. Intelligence is a purely physical phenomenon, which means it can be emulated by computers. Natural brains are also a product of evolution, which means (1) the development happens very slowly, (2) development is directed only towards evolutionary success and (3) there is no flexibility in how the thinking organs are constructed. Computer intelligence does not in principle have these limitations. It would be terribly anthropocentric to believe that humans are the most sophisticated intelligent entity that can exist in the physical world - after all, we are as far as we know the first such entity to emerge, so from our perspective the evolution of intelligence has now stopped.
That's the feasability argument. The risk argument is that the consequences of an independent, runaway intelligent entity significantly more capable than humans would have such devastating consequences for humanity's future that even a small risk merits a significant effort to map out the territory. Respected scientists have said "it is impossible" to hundreds of things that proved to be quite simple, so this is not an argument. Even if you don't buy the previous part of my reasoning, there is still risk here. In principle, there are any number of things that could preclude advanced AI in the near future even if the above reasoning is correct (too difficult, requires too much computing power, uses different computational techniques), but seeing as we don't know the unknowns here, taking the cautious route is the correct thing to do. This has been a scientific principle for decades, no reason to drop now.
Do AI researchers have any arguments opposing this that don't amount to "the AI we have created up until now is not very good"?