Re:What If the Singularity Does Not Happen? (2007) [video]
by TheOtherHobbes on Friday August 16, 2019 @05:12AM
(#20713413)
There's a more basic problem - the Singularity assumes automated self-improvement is possible.
The reality is that most codebases are brittle and bug-ridden. Poor reliability is one the defining characteristics of software systems.
The most reliable systems require formal methods, which require a nailed-down spec. There is no nailed-down spec for an open-ended self-improving system, and it's debatable if a spec is even possible.
So the only practical systems will be heuristic and inherently unreliable in at least some domains, with a tendency to fail spectacularly - much like humans.
This is a basic limitation of technology. You can have mechanical perfection in a limited domain (although with software you often you get a poor imitation of that) or you can trade off flexibility for reliability.
The reality is that most codebases are brittle and bug-ridden. Poor reliability is one the defining characteristics of software systems.
The most reliable systems require formal methods, which require a nailed-down spec. There is no nailed-down spec for an open-ended self-improving system, and it's debatable if a spec is even possible.
So the only practical systems will be heuristic and inherently unreliable in at least some domains, with a tendency to fail spectacularly - much like humans.
This is a basic limitation of technology. You can have mechanical perfection in a limited domain (although with software you often you get a poor imitation of that) or you can trade off flexibility for reliability.
You can't have both.