Can we even measure an intelligence greater than our own? And at that point -- what is the point exactly? Why make an intelligence that we don't even know is intelligent? If its output is indiscernible to the human mind, what good does it do us?
The only "barrier" I can see that is valuable and that might be measureable to humans is this underlying mechanic of being able to create new ideas/concepts from little to no existing human generated training data. Otherwise, and as I see it right now, these machines will not become sentient, or go rouge, or be able to cure all future disease or solve all of physics, becasue they are entirely reliant on human made trainingsets.
Historically, for humans, this can not be the case, otherwise we would see no progress. Someone has to have a thought for the first time. Perhaps the solution to this for AI agents is some inherent randomness, and massive amounts of brute force compute. Who knows?
We can always just unplug the computer, you know. AI agents can do damage, they can be used in malicious ways and bring down/distrupt infrastructure, but the leap from today to the point where LLM's can prevent us from pulling the cord is unimagineable large, and likely infinite.
In modernity, the gap between the digital and material world is closing, especially from a sociological perspective, but we are still neurological systems spinning on a large rock. For 99% of our history we didn't have electricity. The likelyhood of returning to a post-electricty world, for example, is much greater than LLM's taking over the world and making us into zoo animals.