The Chinese room confusion shows how effective our deepest biases are at directing our rational thought into whatever level of nonsense you can put up with.
A similar case I noticed is the resistance against the Everett interpretation of the quantum measurement problem (i.e. the wave function collapse). For some people, it's so hard to accept the consequences of your own consciousness being an emergent phenomenon of matter that interacts with other matter and thus it partakes in the very fundamental effects of such interactions, including entanglement.
Re:This Week's Hype
by jeffparsons on Friday December 02, 2022 @07:16AM
(#33829551)
I think you've understood the idea perfectly, including all the ways in which Searle's argument is rubbish.
However, I come to a different conclusion. I think that all this is exactly why the Chinese room concept is interesting — because it is so obvious, but somehow induces a huge resistance in so many people. I.e. the thought experiment itself isn't all that interesting — people's reactions to it are.
So, the opposite assumption is that there's nothing more to intelligence than a "mere computation", and all that stuff like inner dialog and "a sense of self" are just side-effects of it. That way, there's no difference between "strong AI" and "weak AI" in his terminology, no difference between "understanding" and "simulation of understanding". Then, of course, a digital computer has a mind and understands Chinese, and also the Chinese room has a mind and understands Chinese. But Searle doesn't understand Chinese and this is supposed to be a contradiction.
But there's no contradiction here, because Searle is not the Chinese room. He isn't the mind, in this model he is to Chinese room what electricity is to a digital computer, or what neurotransmitters are to human brain. So, yeah, electricity doesn't understand Chinese, serotonin doesn't understand Chinese, single human neuron doesn't understand Chinese, but Turing-test-passing computer, Chinese brain and Chinese room all do. Searle is just a detail in the Chinese room — tiny and insignificant compared to a book, containing a complete description of a program that speaks Chinese and is able to form coherent (and persistent!) opinions on every possible topic in the world.
So, the fallacy is equating Searle's role in the hypothetical Chinese room to the whole digital computer that speaks Chinese and passes Turing test with breeze.