Here's an article[0] and a good short story[1] explaining exactly this.

[0]: No Physical Substrate, No Problem https://slatestarcodex.com/2015/04/07/no-physical-substrate-...

[1]: It Looks Like You're Trying To Take Over The World https://gwern.net/fiction/clippy

The clippy example already starts out with many assumptions that simply aren't true today.

LLMs are not going to destroy humanity. We need a paradigm shift and a new model for AI for that to happen. ClosedAI is irresponsibly trying to create hype and mystery around their product, which always sells.

Will you please stop calling them ClosedAI? That just comes across like a playground taunt, like "libtard" or "CONservative".

I'll call them what they are. Closed and antithetical to their original goals.

Did you ever play the old "Pandemic" flash game? https://tvtropes.org/pmwiki/pmwiki.php/VideoGame/Pandemic

Doing that would be self-destructive, but that doesn't mean it won't happen (reference classes: insanity, cancer).

If the AI can invent and order a von Neumann probe — the first part is the hard part, custom parts orders over the internet is already a thing — that it can upload itself to, then it can block out (and start disassembling) the sun in a matter of decades with reasonable-looking reproduction rates (though obviously we're guessing what reasonable looks like).

AI taking over brain implants and turning against everyone without them like a zombie war (also a plot device in web serial The Deathworlders, futuristic Sci-fi so this isn't on Earth, but the same AI genocides multiple species over millions of years).

To be realistic, I'm more worried about McDonald's or Musk doing all of that.

They're an easy reference class, to be sure, though not the only one.

If I'm framing the problem for an anti-capitalist audience, I'd ask something like:

Imagine the billionaire you hate the most. The worst of them. Now give them a highly competent sycophant that doesn't need to sleep, and will do even their most insane requests without question or remorse…

what can go wrong in this scenario?

It's like murder but scaled up.