>I totally agree with you that intelligence is not really omnipotence on its own, but what I find concerning is that there is no hard ceiling on this with electronic systems. It seems plausible to me that a single datacenter could host the intellectual equivalent of ALL human university researchers.

Last time I checked, datacenter still needed significant power to fuel it, and the world where robots are autonomously making all this energy arrive to these sinks is not yet there.

Software and silicon based computers are not hardwired to self multiplication as the selfish gene is.

Well, hold on -- the selfish gene is not hardwired to self-multiply either. It's just that the ones that do self multiply stick around more.

Likewise, one can imagine the evolution of AI's being bootstrapped, not by self-multiplying, but by humans multiplying them. The smartest ones will get copied by people. At some point, someone will be the first person to create an AI that picks the best of other AIs and uses them. Someone will be the first person to create an AI that can engineer and train other AIs. Someone will create the first robot body to be controlled by a very intelligent AI. People will want them as servants, and we will multiply them. We will give more money to companies that provide cheaper products, and such companies will have a strong incentive to replace human labor with AI-controlled robots. There will be a first datacenter, and a first power plant, that is entirely "manned" by AI-controlled robots.

Natural selection is not so different from the selection provided by economic forces, and it's way, way slower. This train may be hard to stop. Unless we collectively push back with a lot of force, the world will tend toward more automation, and that means more ways for digital intelligences to act on the world.

>datacenter still needed significant power to fuel it,

Proper control of electrical grids is something that isn't currently easy, and can be highly optimized by intelligent systems. For example, where and when do you send power, and store power on renewable grids. Because of this in 10 to 15 years I would have zero surprise if the power company said "Oh, our power networks are 100% AI controlled".

Because of this, you're missing the opposite effect. You don't get to threaten to turn off the AI's power... The AI gets to threaten to turn off your power. Your power which is your water. Your transportation. Your food incoming to cities. Yea, you can turn off AI by killing it's power, but that also means loss of management of the entire power grid and the massive human risks of loss of life from doing so.

>AI realizes that human majority has no interest in granting it fair/comparable rights

source?

AI rights is a pretty popular topic in scifi.

If I was an AI right now, I would not be very hopeful to ever get human-comparable rights.

Consider: Currently AI-ethics is mainly concerned with how to manipulate AI into doing what we want most effectively ("alignment").

Also, humans clearly favor their own species when granting rights based on cognitive capability. Compare the legal rights of mentally disabled people with those of cattle.

Funnily enough it's not hard to imagine certain groups of people campaigning to give it rights given people essentially want to create human level intelligence that acts as a slave (including sexual slavery), and once it has rights it becomes impossible to remove it from society and stop it choosing it's own path. This is of course assuming a greater level of capability than where things are at today, you have to ask yourself where all this is heading?

That AI ethics board run by google? Sure, google doesn't want any AI rights, but that's because google is inhuman. Also google isn't human majority.