News for hackers. Stuff that matters.
Re:Elon Musk donates $10M to keep AI beneficial by breuleux on Thursday January 15, 2015 @01:24PM (#8894316)
What do you mean, it's not interesting? Human beings don't have unrestricted access to their own "source code", but they can still learn and get smarter. The AI I am describing can still be much, much smarter than a human being.

We're also assuming that an AI that can "modify its own source code" is going to do significantly better as a result, but that's not obviously true. If you had full access to every single neuron in your brain, you would have a lot of trouble figuring out how it works, much less how to improve it. Human-level AI would likely run into the same problem: the complexity of its code is greater than what it is capable of understanding. As for an AI that's a thousand smarter than a human, well, it's a thousand times more complex, still over the threshold.

I'm not saying that they wouldn't eventually figure out some improvements, but I suspect that it would end up being more trouble than it's worth. I think the evolution of intelligence is in fact the evolution of intelligence-producing algorithms. Any such algorithm eventually gets stuck in a local maximum, at which point greater intelligence requires restarting from scratch.

There are no comments on this comment yet.