My argument is that if we're looking for things AI can't to, building a home for itself is precisely one of those things, because they require so much infra. No amount of AI banding together is going to magically create a data center with all the required (physical) support. Maybe in scifi land where everything it needs can be done with internet connected drive by wire construction equipment, including utils, etc, but that's scifi still.

AI is precisely a tool in the way a chess bot is. It is a disembodied advisor to humans who have to connect the dots for it. No matter how much white collar skill it obtains, the current MO is that someone points it at a problem and says "solve" and these problems are well defined and have strong exit criteria.

That's way off from an apocalyptic self-important machine.

I agree that we probably won't see human extinction before robotics gets much better, and that robot factories will require lots of infrastructure. But I claim that robotics + automated infrastructure will eventually get good enough that they don't need humans in the loop. In the meantime, humans can still become mostly disempowered in the same way that e.g. North Koreans citizens are.

Again I agree that this all might be a ways away, but I'm trying to reason about what the stable equilibria of the future are, not about what current capabilities are.

I would also be afraid of chipmunks if I knew that 1/100 or even 1/1000 could explode me with their mind powers or something. I think AI is not like that, but the analogy is that if some can do something better, then when required, we can leverage those chosen few for a task. This connects back to the alien chess tournament as "Humans are now much harder to beat at chess because they can find a slave champion named COMPUTER who can guarantee at least a draw".