Re:Apocalypse soon: the scientists preparing for the end times
by TeMPOraL on Tuesday September 30, 2014 @08:21AM
(#8387746)
It's a complex topic but the primary danger is with self-improving AIs. That pretty much implies there is going to be only one - the first one that starts to improve itself to be smarter than us. So it will look exactly "that our software will be progressively getting smarter, and our databases larger" up until the point some kind of critical mass is reached, and after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later.
You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you). But there is no reason why this has to happen, and even if we did succeed in copying exact human values to a human-level AI, we'll likely be soon locked in an inter-species conflict over access to resources - after all, we don't care about other species living on Earth, so why the AI should care about us?
You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you). But there is no reason why this has to happen, and even if we did succeed in copying exact human values to a human-level AI, we'll likely be soon locked in an inter-species conflict over access to resources - after all, we don't care about other species living on Earth, so why the AI should care about us?