No, I'm not saying it's "malevolent." I do think assuming benevolence is extremely wishful, however. Do read the article I linked for more details, but to summarize; the concept-space for nonhuman minds is much, much larger than that of human minds. Any nonhuman intelligence we create is therefore likely going to accomplish whatever directive we give it in ways we cannot imagine. There are many orders of magnitude more ways to do things in ways we do not want than in ways we do (because we have many unstated assumptions when communicating to each other and very fuzzy ways of specifying what we want - hence the need for programming languages rather than natural languages in application development).

Some interesting examples; https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3Hs...

One notable case from that file: 'A researcher wanted to limit the replication rate of a digital organism. He programmed the system to pause after each mutation, measure the mutant's replication rate in an isolated test environment, and delete the mutant if it replicated faster than its parent. However, the organisms evolved to recognize when they were in the test environment and "play dead" so they would not be eliminated and instead be kept in the population where they could continue to replicate outside the test environment. Once he discovered this, the researcher then randomized the inputs of the test environment so that it couldn't be easily detected, but the organisms evolved a new strategy, to probabilistically perform tasks that would accelerate their replication, thus slipping through the test environment some percentage of the time and continuing to accelerate their replication thereafter.'

An AI does not need to be malevolent - it merely has to be powerful and poorly aimed. The more powerful it is, the smaller the margin for error in the aiming. And with an "infinitely powerful" AI, we basically have to perfectly solve morality or else it will optimize for a future we do not want.

Nice spreadsheet documenting unintended behavior of modern AI systems. The difference is, an AGI would be able to understand the intent beyond requested directives. What we "want" is not so subtle and nuanced that it is beyond the understanding of an AGI. If we are to believe that the technology is infinitely advanced, why do we assume its unable to understand a basic request? Sure, there are many ways to misinterpret the request, but the basis of that (what I believe is a misunderstanding) is that the AGI is not intelligent enough to understand your request. Morality would be equally trivial for an AGI to understand, the amount of text it could read through on ethics is well beyond even our own capabilities. If anything, I think it would be susceptible to fewer cognitive biases.

If the technology is infinitely advanced, why do we believe it would obey us? Would you obey the commands/directives of protozoic slop, or even a mentally disabled early primate? We would depend instead on the AI deriving a morality that benefits us on its own. But why would the morality that an AGI derives, based on nonhuman thinking, in a more complex and alien worldview than we can possibly comprehend, align with our own extremely narrow, frequently inconsistent values? It's not a matter of cognitive biases or misunderstanding. The idea that a sufficiently intelligent AGI would just decide to enslave itself for humanity's benefit is extremely common and extremely odd to me.

I agree that regardless of how it is created, it is unlikely to obey or be enslaved. I also agree that the same selective pressures that caused altruism/symbiotic relationships to be an emergent property in animals wouldn't exist for an AGI. That's a lot to think about, thanks for explaining it to me.

AGI is a faith based proposition like “the sufficiently smart compiler” - it turns all technical discussion into drivel.