Sorry, I just wasted ten minutes watching a YouTube video that someone else in this thread asserted was going to explain everything. Said video was entirely tangential to the point in question - why can't we just tell the AGI to sod off with it's wishes and goals. At the end of the day it is all just bits going through wires, which we are perfectly capable of disconnecting.

Now I'm not going to spend a couple of hours watching a film that has a mediocre IMDB rating. But I do notice that the synopsis says "they gave the AGI total control over the US nuclear weapons arsenal".

Yeah, let's not do that, I agree fully on this point. If we are concerned a thing will be evil and/or coercive, let's not give it weapons or other means by which to enact coercion.

This is kind of my point - the AGI won't just stumble over the nuclear codes in a Reddit thread. For the AGI to actually accomplish something of real-world relevance, us humans have to agree.

> Yeah, let's not do that, I agree fully on this point. If we are concerned a thing will be evil and/or coercive, let's not give it weapons or other means by which to enact coercion.

The problem is that we can't know if the AGI is behaving maliciously or not far into the future, because it has potantial to be far more intelligent than us.

As for coercion, let's think of it as a manipulation. If a super intelligent agent has malicious goals, it can be very manipulative and be subtle in the process as to not spook humans. Companionship of intelligence with evilness is scary.

Of course we can (if we won't have distributed machines running AGI, because things like this may complicate things) "unplug" it. But the real problem arises from the fact that us humans can be easily manipulated. That's how I look at the problem.

> Now I'm not going to spend a couple of hours watching a film that has a mediocre IMDB rating.

your loss

> For the AGI to actually accomplish something of real-world relevance, us humans have to agree.

Donald Trump managed to get control of the US nuclear arsenal

Yes, now we are getting somewhere. Donald Trump managed to somehow get control in a very limited sense of the US nuclear arsenal.

First of all, how did he get this control? Certainly it was not via his profound intelligence, but because other humans chose to give him control.

Second of all, we know beyond a shadow of a doubt, that if Donald Trump had ordered a nuclear strike in a situation where it was not consistent with US nuclear weapons doctrine, his command would have been disobeyed and the 25th amendment would have been enacted poste haste.

So again, how would an AGI be able to get into office or into another position of power? And how would it be able to order a nuclear strike in a situation inconsistent with nuclear doctrine and not be overriden by those lower in the chain of command? Why would there be different rules applying to the AGI as compared to the human?

> Second of all, we know beyond a shadow of a doubt, that if Donald Trump had ordered a nuclear strike in a situation where it was not consistent with US nuclear weapons doctrine, his command would have been disobeyed and the 25th amendment would have been enacted poste haste.

how do you know this beyond a shadow of a doubt?

your entire "argument" is based on arrogant assumption after arrogant assumption