* "Jack of all trades, master of none" general models like GPT and other LLMs, or like diffusion image generator models
* Hyper-focussed expert-to-superhuman performance models like AlphaZero (beats all humans), Cicero (Facebook's Diplomacy player, ranked top 10%), Pluribus (Facebook's no-limit Texas hold 'em poker player), and the one whose name I forget that learns how to map WiFi interference into pose estimation so well it can be used for heart rate/breathing sensing, etc.
And of course, we've got people who say "this can't possibly fail!" who then take a model they don't understand, put it in a loop, give it some money, and then it does something unexpected. Mostly this only results in small-scale problems, but all automation so far has failure modes[0] and there's no reason to presume this trend won't continue even when it's as smart as a human.
If it gets smarter than a human, it might still make actual objective mistakes, but we also have to consider that, within the frame of reference of it's goals[1] it may be perfect, and yet those goals just aren't compatible with our goals, perhaps neither as individuals nor as societies nor as a species.
[0] my favourite example, Cold War, Thule airforce base early warning radar reports a massive radar signature coming over the horizon. All indications are a massive Soviet first-strike!
The operators forgot to tell the system that the Moon wasn't supposed to respond to an IFF ping.
[1] regardless of whether those goals are self-made or imposed from outside as a result of how we as humans construct its rewards, before anyone asks about robot free will or whatever, as the cause of those goals doesn't matter in this case
But we're not being scientific. We're over-indexing on wild imagination.
This game of "what if" is being played by everyone except for the stakeholders that are telling you to slow down and listen to what's actually being built. Laymen and sci-fi daydreamers are selling fears and piling up hurdles that do not match the problem.
In any case, none of this hand-wringing will actually result in tangible "extinction prevention" regulation. So I'm not worried on that front. The concern is that this gives regulators a smoke screen to make it harder for small companies to compete. And that's what's actually happening right now.