i.e. the AI won't come up with the goals itself, we cause its goals whatever they happen to be, those goals are different from the ones we wanted, we remain essentially ignorant of the difference between what we said and what we meant until after it goes wrong.
This happens at basically every scale, so we've already seen it in toy model AI before the invention of the Transformer models or even considered as many as one thousand parameters.
Large models still go wrong, they just happen to go wrong with more complext tasks. We had to figure out how to make them not-wrong with the smaller ones (like coding) to make them capable of bigger errors (like hacking out of their sandbox).
From your position, catastrophes will be the fault of whoever was dumb enough to put a probabilistic system in charge of critical infrastructure. Totally different fear than runaway intelligence and takeover.
The AI labs use reinforcement learning to train LLMs, and have spent a while now giving each new generation of LLM an increasing role in training the next one.
"AI psychosis" is also essentially a reinforcement loop, on how much LLM output makes users feel warm fuzzies about how competent the AI is, which was a proxy for (and yet actually different to) how competent the AI really is.
(Aside: "a probabilistic system in charge of critical infrastructure" was never not true, this is one of those phrases people use rhetorically, but it's kinda a thought-terminating cliché given how often human thought randomly fails us; the difference is which things fail how often, not that it happens at all).