All it takes is for someone to give an AI that thinks 100 times faster than humans and overly broad goal. Then the only way to counteract it is with another AI with overly broad goals.
And you can't tell it to stop and wait for humans to check it's decisions, because while it is waiting for you to come back from your lunch break to try to figure out what it is asking, the competitor's AI did the equivalent of a week of work.
So then even if at some level people are "in control" of the AIs, practically speaking they are spectators.
And there is no way you will be able to prevent all people from creating fully autonomous lifelike AI with its own goals and instincts. Combine that with hyperspeed and you are truly at it's mercy.
It's got a massive new investment and research focus, is a very specific application, and room for improvement in AI model, software, and hardware.
Even if we have to "cheat" to get to 100 times performance in less than five years the effect will be the same. For example, there might be a way to accelerate something like the Tree of Thoughts in hardware. So if the hardware can't actually speed up by that much, the effectiveness of the system still has increased greatly.
I say this as a "doomer" who buys the whole argument about AI X-risk.
No human can do that, the system is here, and so is an architecture.
As for the existential risk, assume nothing other than evil humans will use it to do evil human stuff. Most technology iteratively gets better, so there's no big leaps of imagination required to imagine that we're equipping bad humans with super-human, super-intelligent assistants.