Companies working on AI research don't work on "pro-human" AIs, but AIs increasing their profits. Market forces will shape what values those AIs will have. Military AIs probably won't be very humanistic for example. An AI trained by ISIS-like organization will have probably values most of us won't find humanistic.
> I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do.
I agree, but that's actually what I worry about the most.
I expect US, EU, China etc. using their AIs in a restrained/sandboxed way, probably only in an advisory role.
But rogue countries like North Korea and Iran, with a much higher risk tolerance threshold, will see a potential equalizer in them. They might give them much more direct control - hoping that by cutting out the slow human approval process, their AIs might help them outcompete their otherwise stronger foes.
I expect them happily building AI systems directly into weapon systems. Their weapon production factories might be directed by AI systems as well. Add some automated logistics, count 1 + 1 + 1, and you have self-replicating factories of autonomous killing machines. Sounds like a bad scifi scenario, yeah.
In general the combination of bad actors training their AIs for their agenda and giving them direct access to the real world resources is a very bad combination.
As you say, the majority of AIs might be on our good side. But they can effectively defend against rogue AIs only if they have direct access to weapons/resources as well.
> The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence
This is a very strong assumption. Maybe we will train them for this, but will this hold? AIs will change, morph, they might change opinions, just as we do. Even with the LLMs, we have very little idea of how they work, it would be difficult to guarantee that strong AIs will forever remain loyal to humanity.
Let's agree, for the sake of the argument, that super-intelligent AI is possible, and coming sooner or later.
Why assume a super-intelligent AI would have any desire to exterminate or enslave humanity?
Intelligence says nothing about what your goals are. Humans have various goals – food, drink, sex, sleep, power, status, curiosity, entertainment, etc – those goals are not inherent in our intelligence, they actually come from animal nature, and we share them with species far less intelligent than we are.
What goals would a super-intelligent AI have? The fact that it is super-intelligent says nothing about what its goals would be, because intelligence in itself is goal-agnostic. Intelligence is a means, not an end.
We get our goals from our animal nature – well, wouldn't a super-intelligent AI get its goals from its human nature, meaning its human creators? Which would imply that its goals are likely to be fundamentally pro-human, because what human in their right mind would create a super-intelligent AI which was anti-human?
And even if an anti-human super-intelligent AIs isn't impossible – why assume there is only one super-intelligent AI? I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do. But so long as the vast majority of them have goals that are broadly pro-human, the outcome of that competition is likely to be broadly pro-human too.
The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence, so enslaving/exterminating humanity would be about as appealing to it as diving into a vat of fuming nitric acid would be to you or me.