In my estimation, the likelihood of any AGI we develop actually doing that - in the improbable circumstance we create one before destroying ourselves as a species through ecological devastation, nuclear/biological warfare, resource exhaustion, or something even more stupid - is comparable to blindfolding yourself, spinning randomly, then firing a sniper rifle through a winning lottery ticket a kilometer away.

The simplest(!) problem to solve before AGI becomes anything other than a civilizational footgun is "rigorously defining perfect morality." After that comes the task of programming something that we have very fuzzy, often misleading intuitions about (i.e. general intelligence). And programming it perfectly, of course, since any deviation from the defined spec will trigger the footgun. And since the thing will have to be able to self-modify, we have to assume an adversary who is by definition (likely orders of magnitude) more intelligent than you possibly trying to subvert said programming.

I find Yudkowsky's writing rather illuminating in this regard (for example, https://www.greaterwrong.com/posts/4ARaTpNX62uaL86j6/the-hid...) although he remains far more optimistic than I am.

We're programmers, considering the creation of what is essentially a god. As if an algorithm merely destroying a life through a bug were unheard of, or too small stakes.

What aspect of an AGI would make it malevolent by default? I lean towards benevolence by default, but I can't think of any pressures either way. The paperclip analogy I read about often seems misguided, why would we assume that a self-improving artificial intelligence would have trouble understanding morality? Giving it directives seems pointless, I doubt there's a way to contain AGI technology. With infinite upside, and the trajectory of our species being gradual extinction (as said in the rest of this thread) I'm optimistic.

No, I'm not saying it's "malevolent." I do think assuming benevolence is extremely wishful, however. Do read the article I linked for more details, but to summarize; the concept-space for nonhuman minds is much, much larger than that of human minds. Any nonhuman intelligence we create is therefore likely going to accomplish whatever directive we give it in ways we cannot imagine. There are many orders of magnitude more ways to do things in ways we do not want than in ways we do (because we have many unstated assumptions when communicating to each other and very fuzzy ways of specifying what we want - hence the need for programming languages rather than natural languages in application development).

Some interesting examples; https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3Hs...

One notable case from that file: 'A researcher wanted to limit the replication rate of a digital organism. He programmed the system to pause after each mutation, measure the mutant's replication rate in an isolated test environment, and delete the mutant if it replicated faster than its parent. However, the organisms evolved to recognize when they were in the test environment and "play dead" so they would not be eliminated and instead be kept in the population where they could continue to replicate outside the test environment. Once he discovered this, the researcher then randomized the inputs of the test environment so that it couldn't be easily detected, but the organisms evolved a new strategy, to probabilistically perform tasks that would accelerate their replication, thus slipping through the test environment some percentage of the time and continuing to accelerate their replication thereafter.'

An AI does not need to be malevolent - it merely has to be powerful and poorly aimed. The more powerful it is, the smaller the margin for error in the aiming. And with an "infinitely powerful" AI, we basically have to perfectly solve morality or else it will optimize for a future we do not want.

Nice spreadsheet documenting unintended behavior of modern AI systems. The difference is, an AGI would be able to understand the intent beyond requested directives. What we "want" is not so subtle and nuanced that it is beyond the understanding of an AGI. If we are to believe that the technology is infinitely advanced, why do we assume its unable to understand a basic request? Sure, there are many ways to misinterpret the request, but the basis of that (what I believe is a misunderstanding) is that the AGI is not intelligent enough to understand your request. Morality would be equally trivial for an AGI to understand, the amount of text it could read through on ethics is well beyond even our own capabilities. If anything, I think it would be susceptible to fewer cognitive biases.

If the technology is infinitely advanced, why do we believe it would obey us? Would you obey the commands/directives of protozoic slop, or even a mentally disabled early primate? We would depend instead on the AI deriving a morality that benefits us on its own. But why would the morality that an AGI derives, based on nonhuman thinking, in a more complex and alien worldview than we can possibly comprehend, align with our own extremely narrow, frequently inconsistent values? It's not a matter of cognitive biases or misunderstanding. The idea that a sufficiently intelligent AGI would just decide to enslave itself for humanity's benefit is extremely common and extremely odd to me.

I agree that regardless of how it is created, it is unlikely to obey or be enslaved. I also agree that the same selective pressures that caused altruism/symbiotic relationships to be an emergent property in animals wouldn't exist for an AGI. That's a lot to think about, thanks for explaining it to me.

AGI is a faith based proposition like “the sufficiently smart compiler” - it turns all technical discussion into drivel.

Every problem is probably hyperbole. It will solve lots of problems. But it feels a bit naive to expect it to solve all problems. We've got so much more technology than say, ancient Greece ever imagined, but we still have problems.

I'm intrigued. What problem would you say is out of reach of an AGI (beyond what is understood to be physically impossible)?

Writing sonnets, directing movies, preventing a 13 year old from taking heroin, preventing police abuse, solving the Israel-Palestine issue, solving depression, figuring out which religion is correct.

Things that it could probably do: Write song lyrics, screenplays, identifying drug dealers, things political parties can agree on, gauging your mental health similar to how Garmin watches can gauge physical health, finding MH 370, pass a typical tech interview.

Also if you play with AI as it stands today, you'll notice some characteristics. It reminds me a little of Vision in WandaVision. It's adorably naive in some things. It has a certain "dialect", probably depending on how it's trained. It reminds me of a 14 year-old American.