Re:The greatest risk of AI is from the people who control it...
by SpicyLemonZest on Wednesday May 31, 2023 @04:30PM
(#36143901)
I don't think it can be reasonably disputed that software re-deploying an improved version of itself is plausible and obvious. Automated commits and automated deployments are both fairly common in modern build systems.
Re:The greatest risk of AI is from the people who control it...
by jazzyjackson on Wednesday May 31, 2023 @05:01PM
(#36144275)
That software exists to automatically apply changes to code doesn't have much bearing on software being able to introspect and decide what changes to apply, but maybe I'm moving the goalposts. The infrastructure exists for programs to edit their own deployment, what I'm doubting is that the software will have any idea of what can be improved without a human-in-the-loop.
Compiler optimizations are a counterexample, could be called automated self-improvement -- but it's to do the same thing with fewer resources, what I'm looking for is a reason to believe an optimization could go the other way, have more capabilities with the same resources.
Re:The greatest risk of AI is from the people who control it...
by mrguyorama on Wednesday May 31, 2023 @05:48PM
(#36144812)
It's important to note that the lesswrong's view of AGI is that it sees everything, knows everything, can do everything, can make humans do anything, and has motives directly harmful to us.
These are all taken as a given because the entire concept is just old testament god but with glowy parts. This is an essential part of the dogma, which is why there's never any sort of justification for it. Super smart computer is just assumed to be magic.
It's plausible and obvious to them because "a super-intelligence can make anyone do anything", can reprogram any computer to it's will, and can handwave away literally technical deficiencies.
> For example, it might program a virus that will infect every computer in the world, causing them to fill their empty memory with partial copies of the superintelligence, which when networked together become full copies of the superintelligence. Now the superintelligence controls every computer in the world, including the ones that target nuclear weapons. At this point it can force humans to bargain with it, and part of that bargain might be enough resources to establish its own industrial base, and then we’re in humans vs. lions territory again.
I believe there's a rift between doomers and eye-rollers because this kind of leap sounds either hollywood-hacker-sci-fi or plausible-and-obvious. The notion that software can re-deploy an improved version of itself without human intervention is just outside the realm of possibility to me (or somehow blackmailing or persuading a human to act alongside it?? Is that AI anymore or is that just a schizophrenic who thinks the computer is talking to him?)