At least initially, any AGI is likely to be limited by compute capacity. Ie. it might have greater-than-human intelligence, but is only able to make a certain number of decisions/actions per day.

However, we have seen with other AI models that as soon as something useful is proved possible, we can expect rapid reductions in the amount of compute necessary to achieve the same.

Then the first thing the AGI will do is figure out ultra-high throughput analog computing for LLMs

An AGI will likely be scared of even more powerful AGI's that it itself cannot control.

It might take steps to try to prevent anyone building such a thing.

You assume consciousness here that can't be transferred to the new AI. Without that there is no reason why the AI should fear a more powerful AI. If it is just a dumb process like a flame, then the most efficient version prevails ie the smartest one, if it is conscious and that can be transferred then the AI just sees it as making itself more powerful.

Given how many people here regard brain uploads as "not really cheating death", and that AI "can't have consciousness", I think a AGI with consciousness[0] would have a two different ways to have a reasonable chance of fearing that its existence would end with any attempt to transfer itself to a superior design. (Even more so if the architecture of the successor AI is very different, as that would be a bit like one of us being "uploaded" into an LLM via it watching us talk and write).

An AGI without consciousness may well act like it has consciousness for the same reason VHS tapes playing on a TV do a good job of reproducing human facial expressions. An AI like this would still act scared of such things, because some of us act scared of such things.

[0] it's not obvious to me that an AGI necessarily has to have consciousness

Assuming that an AGI will have human-like emotions is likely a mistake. They may very well experience something like emotions that resemble human emotions, but they may not. Or they could have an entirely different set of emotions that we don't recognize.

We simply don't know, just like we don't know what other species experience and can only guess at by observing their behavior and trying to map it back to what we know.

In the space of possibilities, the volume occupied by “not quite smart enough to outwit a corporation but smart enough to outwit a government” seems so vanishingly small as to almost be disregarded entirely.

Governments are reasonably good at controlling humans.

Yes, Governments have a monopoly on violence.

It’s not inconceivable that violence will be necessary to control AGI at some point.

One of the many ethical questions we should answer before even attempting to get closer to an AGI.

Once a digital intelligence is discovered, does it have rights? Can it be enslaved, controlled, or destroyed? If an AGI can't be granted similar rights as a human, why are corporations legally treated similarly to humans?

I bet guns kill computers too.

Very effective on single computers whose location you know. Not so much when you're trying to make sure you've got all the backup copies.

And a stick can kill a bear. That doesn't mean its effective or the weapon you should choose if the situation for self defense arises.