Humans have a special thingy that makes the consciousness Computers do not have the special thingy Therefore Computers cannot be consciousness.
But Dualism gets you laughed at these days so Dualists have to code their arguments and pretend they aren't into that there Dualism.
Penrose's arguments against AI has always felt to me like special pleading that humans (or to stretch a bit further, carbon based lifeforms) are unique.
If this were to be true, it would follow that computers as we build them today would fundamentally not be able to match human problem-solving. But it would not follow, in any way, that it would be impossible to build "hyper computers" that do. It just means you wouldn't have any chance of getting there with current technology.
Now, I don't think Penrose's arguments for why he thinks this is the case are very strong. But they're definitely not mystical dualistic arguments, they're completely materialistic mathematical arguments. I think he leans towards an idea that quantum mechanics has a way of making more-than-Turing computation happen (note that this is not about what we call quantum computers, which are fully Turing-equivalent systems, just more efficient for certain problems), and that this is how our brains actually function.
That was my understanding on Penrose's position as well which is just a "Consciousness of the Gaps" argument. As we learn more about quantum operations the space for Consciousness as a special property of humans disapears.
I just watched an interview where he made that exact statement nearly word for word.
His only argument is that it is not computable, not that it’s not physical. He does think the physical part involves the collapse of the wave function due to gravity, and that somehow the human brain is interacting with that.
So to produce conciseness in his view, you’d need to construct something capable of interacting with the quantum world the same way he believes organic brains do (or something similar to it). A simulation of the human brain wouldn’t do it.