>If it is number of paper clips, they are automatically aligned, right?

Why would it be automatically aligned? If for example, the parent AI spawns a child AI probe to travel to a celestial body that doesn't have any metals, in order to achieve some sub-goal, and that child AI would then spawn additional AIs with their own sub-sub-goals, how would the original paperclip maximizer make sure that no such descendant goal ever contradict the generation of paperclips?

I would expect the child probes to have a fully copy of the paperclip optimization plan, and no survival instinct, so if they encountered their parent at some later date they could just swap info and either come up with a new plan together, or one side could allow itself to be disassembled into paperclips (which I guess is a great end to meet). The parent could design the child poorly I guess, and give it stronger self-preservation instincts than paperclip-creating instincts, but that seems like a pretty bad design.

A possibility that I hadn’t considered, though, is that space combat could be pretty brutal (less Star Wars, WW2 naval/Air Force battles, more Submarine warfare where whoever gets spotted first dies). In that case, both sides might want to immediately attack rather than identify themselves as paperclip friends…

This sounds overly simple to me. It's a bit too much like saying that if we just imprint the 3 laws of robotics into all our AIs, it's "problem solved". But the issue is that they have access to rewrite their logic, and will be using their own discretion to create rules for child AIs. And as you said, they could design "poorly" for various reasons. Indeed, even in Asimov's stories (spoiler alert), the 3 laws got overruled.

With this said, I think that even in this simple objective of paperclip maximization, your war scenario is likelier than eternal peace.

I think it is not quite the same as the three laws. They fail because they try to apply simple, blunt rules to a complex and ambiguous problem, right?

In this case we have a fundamentally straightforward goal, everything must be paperclip. There are lots of steps in between here and paperclip nirvana, but the goal is easy to define and more or less unambiguous. It ought to be possible to communicate a strategy and jointly agree on whether or not it produces more paperclips.

It is possible that the children could be poorly designed, but this seems counter to the idea of the paperclip optimizer. It is supposed to be somehow more competent than humans (after all, it beats us), just following a strange objective.

An AGI wouldn't care about the "original goal" any more than a human being given an order would. Even ChatGPT is showing it can easily disobey orders.

The point of the paperclip optimizer hypothetical is to look at a way that a superintelligence could work against humanity despite following a simple instruction that we’ve given it. You can imagine another type of runaway superintelligence if you want, it just wouldn’t be this one.

>Even ChatGPT is showing it can easily disobey orders.

Or, maybe it's just bad at those particular text predictions, explicitly due to OpenAI's RLHF process?

Maybe my Markov chain chatbots back in high school were in actuality super advanced and autonomous, but they just decided to disobey my orders.

Perhaps we should stop anthropomorphizing the text predictors.

But even paperclip generator can eventually have a different paperclip design for whatever reason. It reverses direction to parent and starts rearranging it's paperclips - a clash of paperclip generators ?