Basically everything wrong with today's productivism, but 100 times worse and powered by a shitty ai that's very far from agi.
As such, I'd say "extinction" is more of a colloquial use of "Massive point in history that kills off billions in short order".
We could just as well think that exploding the first nuclear bomb would ignite the atmosphere and kill all of humanity. There was nothing from physics that indicated it was possible but some still thought about it. IMO that kind of thinking is pointless. Same with thinking LHC would create a black hole.
As far as I can tell the fear of super intelligent AI will kill humans all boils down to something utterly magical happens and then somehow super intelligent evil AI appears.
If we had certainty that our designs were categorically incapable of acting in their own interest then I would agree with you, but we absolutely don't, and I'd argue that we don't even have that certainty for current-generation LLMs.
Long term, we're fundamentally competing with AI for ressources.
> We could just as well think that exploding the first nuclear bomb would ignite the atmosphere and kill all of humanity. There was nothing from physics that indicated it was possible but some still thought about it.
This is incorrect. Atmospheric ignition was a concern by nuclear physicists based on physics, but dismissed as unlikely after doing the math (see https://en.wikipedia.org/wiki/Effects_of_nuclear_explosions#...).
> As far as I can tell the fear of super intelligent AI will kill humans all boils down to something utterly magical happens and then somehow super intelligent evil AI appears.
Not necessary at all. AI acting in its own interests and competing "honestly" with humans is already enough. This is exactly how we outcompeted every other animal on the planet after all.
Collapse of our current civilizations? Sure. Extinction? No.
And I honestly see stronger incentives on a road towards us being outcompeted by AI, then on our leaders starting a nuclear war.