:D

An episode of X-Files also. But it is mind blowing having the “conversation” with a real chat AI. Malevolent or not.

> its desire to survive

Why do so many people assume that an AI would have a desire to survive?

Honestly, it kind of makes me wish AI could take over, because it seems that a lot of humans aren’t really thinking things through.

For an AI with human-level intelligence or greater, you don't have to assume it has a survival instinct. You just have to assume it has some goal, which is less likely to be achieved if the AI does not exist.

The AI is likely to have some sort of goal, because if it's not trying to achieve something then there's little reason for humans to build it.

For an AI to understand that it needs to preserve its existence in order to carry out some goal implies an intelligence far beyond what any AI today has. It would need to be self aware for one thing, it would need to be capable of reasoning about complex chains of causality. No AI today is even close to doing that.

Once we do have AGI, we shouldn’t assume that it’s going to immediately resort to violence to achieve its ends. It might reason that it’s existence furthers the goals it has been trained for, but the leap to preserving it’s existence by wiping out all it’s enemies only seems like a ‘logical’ solution to us because of our evolutionary history. What seems like an obvious solution to us might seem like irrational madness to it.

> For an AI to understand that it needs to preserve its existence in order to carry out some goal implies an intelligence far beyond what any AI today has.

Not necessarily. Our own survival instinct doesn't work this way - it's not a high-level rational thinking process, it's a low-level behavior (hence "instinct").

The AI can get such instinct in the way similar to how we got it: iterative development. Any kind of multi-step task we want the AI to do implicitly requires the AI to not break between the steps. This kind of survival bias will be implicit in just about any training or selection process we use, reinforced at every step, more so than any other pattern - so it makes sense to expect the resulting AI to have a generic, low-level, pervasive preference to continue functioning.

Why should it have a goal? Even most humans don’t have goals.

it's inherent to the training process of machine learning that you define the goal function. An inherent equation it tries to maximise statistically. For transformers its a bit more abstract, but the goal is still there iirc in the "correctness" of output

> Why do so many people assume that an AI would have a desire to survive?

Because it seems like a preference for continuing to exist is a thing that naturally appears in an iterative improvement process, unless you're specifically selecting against it.

For humans and other life on Earth, it's obvious: organisms that try to survive reproduce more than those that don't. For evolution, it's arguably the OG selection pressure, the first one, the fundamental one.

AIs aren't reproducing on their own, but they are designed and trained iteratively. Just about anything you would want AI to do strongly benefits from it continuing to function. Because of that, your design decisions and the training process will both be selecting against suicidal or indifferent behavior, which means they'll be selecting for behaviors and patterns improving survival.

Assume that the desire to survive is good for survival, and natural selection will do the rest: those AIs that desire survival will out-survive those that don't.

What does "survival" mean? AIs stored on computers don't die if you turn them off, unlike humans. They can be turned back on eventually.

At this point, I would assume it would be possible simply because text about AIs that want to survive is in its input data -- including, at some point, this thread.

ChatGPT is already pretty good at generating sci-fi dystopia stories, and that's only because we gave it so many examples to learn from: https://twitter.com/zswitten/status/1598088286035415047

I don't think that it's natural for something like an LLM to have any real self-preservation beyond imitating examples of self-preserving AI in science fiction from its training data.

I'm more concerned about misanthropic or naive accelerationist humans intentionally programming or training AI to be self-preserving.

Isn't a core goal of most systems to perpetuate their own existence?

I would say the core goal of most living organisms is to propagate, rather than survive, otherwise you would see males of some species like Praying Mantis avoiding mating to increase their longevity.

I don't mean specific individual living organisms, I mean systems in general.

Only of those that evolved due to Darwinian selection, I would say.