I assume you already know about the concept of "commitment" in Philosophy, but I'll explain it briefly here in case anyone else comes across this comment.
Sometimes while you can't show that a fact is categorically false (given some available evidence), you can show that people who believe that fact are committed to believing other facts. In this case Lucas' own argument has committed him to the "personal opinion" that an inconsistent machine cannot be an adequate representation of mind. But his belief isn't entirely unfalsifiable. If new evidence makes him give up this belief (e.g. a rise of paraconsistent logics) he'll have to give up his original belief along with it.
Human minds are clearly not consistent. Human minds are a loose collection of cobbled-together heuristics.
Logic is one of those heuristics. It has limits because ultimately "logic" ends up being not a personal opinion, but a personal experience of the truth - or otherwise - of a proposition.
Developed and trained minds have generated a heuristic of independent consensus about truths and experience, and strategies for increasing the correlation between the truth experience and the reality experience.
Less developed untrained minds have a much more naive and subjective experience of "truth."
But both work on the same foundation - a nebulous and indefinable subjective experience that is associated with truth, and is typically mistaken as the definition of it.
It's impossible to "prove" truth, because the experience of deciding whether or not something is true is absolutely subjective.
Any attempt to make an objective definition will always be filtered through that subjectivity and reduced to either having that experience or not having it.
We're back to qualia, as usual. No one knows what "having an experience" really means. Until that question is answered, there are no tools for dealing with the rest.
You can argue "But what about the scientific method, and mathematical proofs?" Same answer - they're practices that generate the truth experience in trained minds. They have a better predictive record and are better at finding consistent patterns than the disjointed heuristics of untrained minds. But they still don't - and can't - prove "truth" in any kind of ultimate definitive guaranteed-to-be-objective way.
They certainly can't guarantee that the truth experience is never biased, contingent, limited, naive, circumstantial, or possibly even just plain wrong.
The implications for AI are obvious. If a machine is better at finding consistent patterns, it will appear smarter than a human. It will still be smarter even if it makes errors that are obvious to a human, because at the same time it won't be making errors that humans do make, and which are similarly "obviously wrong" to the AI.
It's the hit rate that matters, not idealised philosophical perfection.
Think of it as analogous to all of the mathematical quasi-proofs that assume certain conjectures are true. They can often be valuable steps to an actual proof, as we saw with Fermat's theorem.
In this particular case, though, it seems close to circularity - if the primary purpose of the argument is to refute the computational theory of mind, then ruling out a whole class of machines a priori looks to me like begging a chunk of the question - though I appreciate that it leads to a better-defined question.
Anyway, there is a manifold in philosophic space in which proving things to win arguments matters and a manifold where torturing philosophy with the scientific method matters and a manifold in which there is doubt about the external world and one in which confusion arises from lack of rigor regarding language and one in which the goal is to prove the existence of gods and a manifold of other manifolds.
We all get to take our pick because philosophy is a big tent that is big enough that a philosopher is free to wander among the lion tamers and clowns and acrobats and look at what is interesting. Radical skepticism is a useful intellectual tool, but so are Platonic idealism and deconstruction and Kantian metaphysics. Anyone who thinks they've got it all correct is probably ignoring human fallibility (but maybe not).
Which would obviously be false. There are clearly true theorems we will never be able to prove. Simply because we can prove many theorems that simpler formal systems cannot, entails nothing about our logical consistency or completeness.
Now valid arguments can be interesting in their own right even if their premises aren't true.
Given this, the proper way to read Lucas's opinion is: I am taking it as given that the computational theory of mind requires a consistent system of inference rules + math; if that isn't true, my argument might be unsound but not invalid (and its validity is interesting); but I also think it is true for x,y,z reasons (though these additional reasons do not constitute a proof).
In other words, the Lucas-Penrose argument is interesting from both a truth-of-premises and validity perspective which is far from dead in the water.
(That's not intended as an endorsement of Lucas-Penrose. There are deeper flaws in the argument, like the assumption that the truth of the knowledge of the mind implies that the mind KNOWS its own truthfulness.)
The Gettier problem: https://en.m.wikipedia.org/wiki/Gettier_problem
To me, this seems to leave Lucas with the burden of proof, and until he or someone else can counter this objection with something more than an opinion, the argument is dead in the water. I used to wonder why philosophy would hang an elaborate argument on personal opinions, but I have come to appreciate that whenever you speculate about matters where there is insufficient evidence to settle the issue, it always comes down to opinions, though this is often overlooked in the discussion.