Re:Experts who say we shouldn't worry about superintelligent...
by PeterisP on Tuesday October 08, 2019 @01:11PM
(#21194481)
The expected "missing step" is as follows:
1. You start with "if I take some numerical representation of an image/text/sound and I multiply/add/perform non-linear function them through these series of numbers it will map to a number which I can interpret as something meaningful to me" and observe that it's not giving you a sufficiently good solution to a particular goal X that you really want to achieve.
2. You figure out that this is a dead-end, you're stuck, and you're not going to get where you want this way by implementing a passive analysis module. Instead, you try to implement an active self-modifying system to achieve that goal X by analyzing itself and improving itself to be more effective than anything which you could implement directly yourself.
3. It's not initially any good at self-improvement, but you keep throwing accumulated research breakthroughs and computing power at it. This likely takes decades.
4. At some point, eventually (if such a thing is possible at all, eventually it's going to happen) it actually does achieve meaningful self-improvement and starts using that computing power not horribly wastefully in a brute force manner as it did just recently, but reasonably well, now giving it more smartness allowing it to implement even more self-improvement on the same hardware.
4B. It also should be expected to easily obtain much more computing power - e.g. it's obvious that anything that's slightly above average human programmer smartness and has a direct use/desire for computing power (as opposed to simply using it for cryptomining) can get some; anything connected to the internet can get the same resources that current scriptkiddies can get by writing a botnet or a ransomware operator can get by extorting a random municipality; there's nothing except total isolation that could prevent it from buying or stealing a few million dollars worth of cloud computing resources to get started.
5. This thing is alive and will kill us all in maximization of its utility function.
1. You start with "if I take some numerical representation of an image/text/sound and I multiply/add/perform non-linear function them through these series of numbers it will map to a number which I can interpret as something meaningful to me" and observe that it's not giving you a sufficiently good solution to a particular goal X that you really want to achieve.
2. You figure out that this is a dead-end, you're stuck, and you're not going to get where you want this way by implementing a passive analysis module. Instead, you try to implement an active self-modifying system to achieve that goal X by analyzing itself and improving itself to be more effective than anything which you could implement directly yourself.
3. It's not initially any good at self-improvement, but you keep throwing accumulated research breakthroughs and computing power at it. This likely takes decades.
4. At some point, eventually (if such a thing is possible at all, eventually it's going to happen) it actually does achieve meaningful self-improvement and starts using that computing power not horribly wastefully in a brute force manner as it did just recently, but reasonably well, now giving it more smartness allowing it to implement even more self-improvement on the same hardware.
4B. It also should be expected to easily obtain much more computing power - e.g. it's obvious that anything that's slightly above average human programmer smartness and has a direct use/desire for computing power (as opposed to simply using it for cryptomining) can get some; anything connected to the internet can get the same resources that current scriptkiddies can get by writing a botnet or a ransomware operator can get by extorting a random municipality; there's nothing except total isolation that could prevent it from buying or stealing a few million dollars worth of cloud computing resources to get started.
5. This thing is alive and will kill us all in maximization of its utility function.