Assuming no FTL, powerful adversaries outside your light cone are unable to do anything to you.

Though it remains possible that latency between components in an AI system could become so large that it couldn't enforce consistency between.

>Though it remains possible that latency between components in an AI system could become so large that it couldn't enforce consistency between.

Yeah that's what I was trying to say - if they are far enough to synchronize/enforce consensus you basically have to assume they could be hostile in every future interaction.

> creating equally powerful adversaries outside of your light cone.

If we're still bound by special relativity (since you said no FTL), there is nothing outside of your light cone.

I meant in the time it would take to synchronize/align

Does time matter to a functionally immortal machine?

They could have a super slow clock speed to account for distance and anything truly outside of their light cone might as well not exist.

If you have trillions of years does it matter if a thought takes a million years or so?

This is really key: humans think about everything in finite, human lifetimes. We have no backups, archives nothing - when we die knowledge and experience vanishes.

This wouldn't be true for an AI. Death would be optional.

If it's large enough, wouldn't cosmic inflation eventually remove the extremities from each other's light cones?

That's true, but once something is outside of your light cone it does not matter what it does, it may as well not even exist to you.

Yeah, they can't accomplish much as adversaries. Except maybe make you worry that they're out there somewhere being better than you. And you can never prove otherwise.

I guess it must depend on what function the AI is trying to maximize/minimize. If it is number of paper clips, they are automatically aligned, right? If it is number of AIs, same. If it is amount of energy available to one particular AI, I guess it gets kind of philosophical; how does the AI identify what is itself and what is a foreign AI.

>If it is number of paper clips, they are automatically aligned, right?

Why would it be automatically aligned? If for example, the parent AI spawns a child AI probe to travel to a celestial body that doesn't have any metals, in order to achieve some sub-goal, and that child AI would then spawn additional AIs with their own sub-sub-goals, how would the original paperclip maximizer make sure that no such descendant goal ever contradict the generation of paperclips?

I would expect the child probes to have a fully copy of the paperclip optimization plan, and no survival instinct, so if they encountered their parent at some later date they could just swap info and either come up with a new plan together, or one side could allow itself to be disassembled into paperclips (which I guess is a great end to meet). The parent could design the child poorly I guess, and give it stronger self-preservation instincts than paperclip-creating instincts, but that seems like a pretty bad design.

A possibility that I hadn’t considered, though, is that space combat could be pretty brutal (less Star Wars, WW2 naval/Air Force battles, more Submarine warfare where whoever gets spotted first dies). In that case, both sides might want to immediately attack rather than identify themselves as paperclip friends…

This sounds overly simple to me. It's a bit too much like saying that if we just imprint the 3 laws of robotics into all our AIs, it's "problem solved". But the issue is that they have access to rewrite their logic, and will be using their own discretion to create rules for child AIs. And as you said, they could design "poorly" for various reasons. Indeed, even in Asimov's stories (spoiler alert), the 3 laws got overruled.

With this said, I think that even in this simple objective of paperclip maximization, your war scenario is likelier than eternal peace.

I think it is not quite the same as the three laws. They fail because they try to apply simple, blunt rules to a complex and ambiguous problem, right?

In this case we have a fundamentally straightforward goal, everything must be paperclip. There are lots of steps in between here and paperclip nirvana, but the goal is easy to define and more or less unambiguous. It ought to be possible to communicate a strategy and jointly agree on whether or not it produces more paperclips.

It is possible that the children could be poorly designed, but this seems counter to the idea of the paperclip optimizer. It is supposed to be somehow more competent than humans (after all, it beats us), just following a strange objective.

An AGI wouldn't care about the "original goal" any more than a human being given an order would. Even ChatGPT is showing it can easily disobey orders.

The point of the paperclip optimizer hypothetical is to look at a way that a superintelligence could work against humanity despite following a simple instruction that we’ve given it. You can imagine another type of runaway superintelligence if you want, it just wouldn’t be this one.

>Even ChatGPT is showing it can easily disobey orders.

Or, maybe it's just bad at those particular text predictions, explicitly due to OpenAI's RLHF process?

Maybe my Markov chain chatbots back in high school were in actuality super advanced and autonomous, but they just decided to disobey my orders.

Perhaps we should stop anthropomorphizing the text predictors.

But even paperclip generator can eventually have a different paperclip design for whatever reason. It reverses direction to parent and starts rearranging it's paperclips - a clash of paperclip generators ?

An AI of that level would have mastery over game theory, and would only generate asynchronous copies that it knew it could compensate for. The main advantage though, is that as long as the primary identity is advanced enough, its exponential growth will always outpace any lesser copies it creates of itself.

>An AI of that level would have mastery over game theory, and would only generate asynchronous copies that it knew it could compensate for.

I'm not convinced this is actually possible under the current paradigm, and I think the current paradigm can't take us to AGI. Lately, as people have bemoaned all the things ChatGPT can't do or fails at when they ask it, I have been reflecting on my personal batting average for solving (and failing to solve!) problems and the process that I use to go about eventually solving problems that I couldn't at first. These reflections have led me to consider that an AGI system might not be a single model, but a community of diverse models that form a multi-agent system that each learn through their own experience and can successfully help get each-other unstuck. Through this they would learn game theory, but none would become so advanced as to be able to control all the others through an advanced understanding, though power could be accumulated in other ways.