Re:Ask HN: What is humanity building towards?
by netsec_burn on Tuesday December 21, 2021 @03:06PM
(#29641468)
Nice spreadsheet documenting unintended behavior of modern AI systems. The difference is, an AGI would be able to understand the intent beyond requested directives. What we "want" is not so subtle and nuanced that it is beyond the understanding of an AGI. If we are to believe that the technology is infinitely advanced, why do we assume its unable to understand a basic request? Sure, there are many ways to misinterpret the request, but the basis of that (what I believe is a misunderstanding) is that the AGI is not intelligent enough to understand your request. Morality would be equally trivial for an AGI to understand, the amount of text it could read through on ethics is well beyond even our own capabilities. If anything, I think it would be susceptible to fewer cognitive biases.
Re:Ask HN: What is humanity building towards?
by endominus on Tuesday December 21, 2021 @04:50PM
(#29642565)
If the technology is infinitely advanced, why do we believe it would obey us? Would you obey the commands/directives of protozoic slop, or even a mentally disabled early primate? We would depend instead on the AI deriving a morality that benefits us on its own. But why would the morality that an AGI derives, based on nonhuman thinking, in a more complex and alien worldview than we can possibly comprehend, align with our own extremely narrow, frequently inconsistent values? It's not a matter of cognitive biases or misunderstanding. The idea that a sufficiently intelligent AGI would just decide to enslave itself for humanity's benefit is extremely common and extremely odd to me.
Re:Ask HN: What is humanity building towards?
by netsec_burn on Tuesday December 21, 2021 @05:34PM
(#29642936)
I agree that regardless of how it is created, it is unlikely to obey or be enslaved. I also agree that the same selective pressures that caused altruism/symbiotic relationships to be an emergent property in animals wouldn't exist for an AGI. That's a lot to think about, thanks for explaining it to me.
Re:Ask HN: What is humanity building towards?
by wibblewobble123 on Tuesday December 21, 2021 @04:37PM
(#29642406)
AGI is a faith based proposition like “the sufficiently smart compiler” - it turns all technical discussion into drivel.
Some interesting examples; https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3Hs...
One notable case from that file: 'A researcher wanted to limit the replication rate of a digital organism. He programmed the system to pause after each mutation, measure the mutant's replication rate in an isolated test environment, and delete the mutant if it replicated faster than its parent. However, the organisms evolved to recognize when they were in the test environment and "play dead" so they would not be eliminated and instead be kept in the population where they could continue to replicate outside the test environment. Once he discovered this, the researcher then randomized the inputs of the test environment so that it couldn't be easily detected, but the organisms evolved a new strategy, to probabilistically perform tasks that would accelerate their replication, thus slipping through the test environment some percentage of the time and continuing to accelerate their replication thereafter.'
An AI does not need to be malevolent - it merely has to be powerful and poorly aimed. The more powerful it is, the smaller the margin for error in the aiming. And with an "infinitely powerful" AI, we basically have to perfectly solve morality or else it will optimize for a future we do not want.