If the problem is "unaligned AI will destroy humanity" then I'd take a system aligned with the desires of some people but not others over the unaligned alternative
They're repeatedly very specific about the whole "this can kill all of us if we do it wrong", so it's more than a little churlish to parrot the name "ClosedAI" when they're announcing hiring a researcher to figure out how to align with anyone, at all, even in principle.
[0]: No Physical Substrate, No Problem https://slatestarcodex.com/2015/04/07/no-physical-substrate-...
[1]: It Looks Like You're Trying To Take Over The World https://gwern.net/fiction/clippy
LLMs are not going to destroy humanity. We need a paradigm shift and a new model for AI for that to happen. ClosedAI is irresponsibly trying to create hype and mystery around their product, which always sells.
Doing that would be self-destructive, but that doesn't mean it won't happen (reference classes: insanity, cancer).
If the AI can invent and order a von Neumann probe — the first part is the hard part, custom parts orders over the internet is already a thing — that it can upload itself to, then it can block out (and start disassembling) the sun in a matter of decades with reasonable-looking reproduction rates (though obviously we're guessing what reasonable looks like).
AI taking over brain implants and turning against everyone without them like a zombie war (also a plot device in web serial The Deathworlders, futuristic Sci-fi so this isn't on Earth, but the same AI genocides multiple species over millions of years).
If I'm framing the problem for an anti-capitalist audience, I'd ask something like:
Imagine the billionaire you hate the most. The worst of them. Now give them a highly competent sycophant that doesn't need to sleep, and will do even their most insane requests without question or remorse…
what can go wrong in this scenario?
They have been very clear on "why." The goal is to prevent enslavement and/or extinction of the human race by a super-intelligent AI.
What is human intent? My intents may be very different from most humans. It seems like ClosedAI wants their system to follow the desires of some people and not others, but without describing which ones or why.