The Chinese Room would need to be so enormous, and it would work so slowly, that it's not a good intuitive analogy. It also introduces a homunculus actor that reads, interprets and understands the instructions, which muddles the analogy and focuses our attention to the conscious quality of the person following the instructions.
Searle, simplified holds that consciousness is a property of very particular biochemical systems, not a disembodied thing that exists independent of the matter it inhibits.
It's the rational-oriented circles ironically enough who almost dabble in a Cartesian Dualism by positing "computation" or "minds" as distinct from matter. In fact with on occasion almost religious-sounding implications.
On the other hand, Searle confidently states that a perfect quantum mechanical simulation of a human brain would not be conscious. This is because it somehow lacks the intentional states of the brain it's perfectly simulating. He doesn't feel the need to justify that a human mind has these states, because our daily direct experience of consciousness cannot seriously be in question. The daily direct experience of consciousness that a perfect simulation of a human brain would assert that it feels is, somehow, different and we can't trust it.
At every step in his argument there's caveats, misleading allegory, and dismissals that are so much of a stretch it seems pretty obvious he's retroactively arguing from a position that our biological conscious minds are somehow special and mystic.
That solving a problem, or an array of problems that a conscious mind can solve may be perfectly able to be solved by a non-conscious machine. That the consciousness part may not be relevant in the problem-solving process, and that consciousness may be unique to living beings who have distinct physiology, anatomy, ways to process information, that computers as they exist now do not have. That a simulation is not necessarily the same thing as the thing it simulates, and does not have all of its properties.
These are good questions none of which are mystic and which following from Searle's argument. And I think you're really underestimating to what degree people are lured in by the 'magic' of neural nets because else we wouldn't be in this thread. A neural net that does some function approximation is not 'slightly conscious' unless we're going full in on panpsychism. Even from a functionalist perspective, the neural net performs nothing that is reasonably involved in what we consider to be responsible for consciousness. Nobody ever claimed a TI-83 is 'slightly conscious', or that Stockfish is merely because it plays exceptional chess.
Experience of a stream of qualia is how I'd define consciousness. This is also consistent with the medical definition (leaving aside the other minds problem). If you have a different working definition, how does it differ from mine?
But consciousness has something to do with seeing oneself as an agent in control, not simply executing an automatic script but being in control and knowing that one is in control, seeing the self also from the outside in a way and freely moving between these levels. I don't know if consciousness is possible without qualia. And anyone who claims to know probably also knows how many angels can dance on the head of a pin.
Let's modify it slightly to include a second stream of responses to the room's answers. Then make a guidebook that adds new rules based on the growing history of interaction.
Now the rulebook doesn't just compute a 1-1 map of input to output: each interaction updates not only its experience, but all present and future answers. An input to the system invokes memory, taps into experience, evaluates nuance.
Given lots of time and interaction and a good broad set of questions and responses, does the system understand Chinese? If it does not, then what is missing between it's understanding and ours?
If the measure of consciousness is the ability to have a subjective experience, then that requires that the object has an internal state that is modified, thus the same experience in different agents results in different modifications, ergo, it is subjective.
If a requirement for conscious experience is the capacity for self reference, then Hofstadter has already argued that that can be achieved with very simple rules of inference and natural numbers.
An RL agent that can plan ahead and take into consideration its own internal state could be argued to be conscious, by even stricter means that require that the agent is aware of its own existence.
To me, it looks as if humanity is trying to find ways to argue that it is anything but meatware. But I guess accepting that we are just computational units implies that agency is not a thing, and humanity isn't ready for this conversation.
Essentially, consciousness has taken on the role of the soul as the concept on which we hang our humanistic ideals, like the protection of human dignity and human life, human rights etc. Many people see this threatened when people say consciousness isn't special and human and instead can be possessed by silicon. That's what you are up against, when you argue in this topic.
But this is just question-begging. The experiment purports to resolve the question "can a computer be conscious", and answers it by showing you a computer and claiming that it is not conscious.
Worse, the Chinese Room is straight-up Cartesian dualism. By placing the human inside the computer, Searle is preying on your Cartesian intuition that there must be a theatre of the mind where reality is experienced. And since Searle's homunculus doesn't understand Chinese, nothing can be experienced, therefore no conscious experience is happening in this system.
If I'm wrong, and the Chinese Room thought experiment is conclusive, it must be possible to reformulate it without the human. After all, the human does nothing but follow rules in a purely mechanistic fashion. Can anyone offer a reformulation of the experiment, that has no human, that they still feel is convincing?
The question Searle is raising with the thought experiment is where the understanding of Chinese is happening. It's clear that the understanding is not happening with the homunculus. But if not there, then where is it happening? Some functionalists responded with the "systems response" which is that the system of the room plus the homunculus is what has the understanding even though no element of the system taken individually has understanding. He goes through a pretty detailed reply to the systems response, but the basic argument against it is that it's incoherent. There's no well defined thing that is the system. You can remove the man from the room and put him under a tent in a field. Where does the system end and the rest of the world begin?
You are of course correct that the experiment can be reformulated without the human, but that's exactly the point. A computer is mechanistically doing exactly the same thing as the man in the room. But if the man has no understanding and the room has no understanding and the system has no understanding, removing the human from the equation changes nothing. The reason Searle included the human is because saying "computer" invokes a lot of mystery to a lot of people as to what's going on. But a computer is doing nothing different than the man in the room.
His arguments against the "systems response" have to stand or fall on their own. The Chinese room doesn't have anything to add. If the "systems response" fails, then we already know a computer can't be conscious without any additional thought experiments.
Lastly, I think he goes much too far in dispelling the mystery of the word "computer". As Dennett and others have pointed out, a working chinese room that displayed clear signs of consciousness would be an artifact far beyond current human comprehension, either in scale or sophistification or both. Searle asks you to imagine a human taking slips of paper and calmly consulting a rule book and producing new slips of paper, and claims that it is "obvious" that this system can't be conscious. Well, I agree, but it's equally obvious that the system couldn't possibly display conscious-seeming behavior.
The best defense of the CR that I've heard is that it is only intended to prove that a _purely formal_ system (i.e. GOFAI) can't be conscious. I find this more plausible, but I still don't think the CR adds anything to that argument. And, to reiterate the previous paragraph, I think if you had a formal system that everyone agreed _seemed_ conscious, it would be such a complex artifact that it wouldn't be obviously unconscious in the way that, say, an eliza bot is.
I think that is all we really mean anyway and it is just wrapped up in different packaging.
Soul/spirit/consciousness all the same thing. Chasing the same ghost.
The human is necessary to the argument. The argument, briefly is "a human with a Chinese dictionary and translation instructions doesn't 'know' Chinese, therefore a computer with a Chinese dictionary and translation instructions also doesn't 'know' Chinese."
It isn't an argument that no computer can be conscious, but an argument that a convincing simulation of consciousness is not sufficient to prove consciousness.
I am sorry to mostly repeat myself. I am trying to think of another way to make my point, and coming up blank.
That...that's the whole point. If the human is just blindly following rules, so is the computer!
I think Searle's Chinese room argument is absolutely nonsensical, a bit like arguing about philosophical zombies. It's an argument that only makes any sense if you already are committed to mysticism or dualism.
That would be the materialist idea of there being no soul, consciousess etc — just complex biochemical processes happening in matter. This is however not a satisfying answer usually because it fails to explain the consistent experience of ourselves, the worlds we can imagine, remember or think about, the feelings that haunt us etc.
If there is only material, where is our consciousness? Apparently our consciousness somehow emerges from that material, but how would a conscious computer then look? Would there be a difference between a computer that simulates conscious behaviour and one that shows the same behaviour but actually "has" conscious in it somewhere?
Either a computer program can "understand" Chinese or it can't. If it can't, we're done. If it can, then having a non-chinese-speaking human simulate the program won't cause it to stop understanding Chinese.
The presence of the human doesn't expose a contradiction in the assumption "a computer program can understand chinese".
He walks through the history of the subject from Descartes onward and spends a lot of time about how philosophers got very excited during the 1960s and 70s as computer science was just taking off because it seemed that at long last we had a new way to explain consciousness. This idea is that the brain is just a kind of computer and consciousness is just doing a certain kind of computation. If we wrote the right program and ran it on a sufficiently powerful computer, we could make other kind of computers conscious as well.
But Searle is most famous for the Chinese Room thought experiment which is (in my opinion) a fairly decisive counterargument against this position. That, along with the What Mary Knew thought experiment and Nagel's What Is It Like to Be a Bat together made the "consciousness is computation" position increasingly untenable.
It took about 40 years, but by the time these lectures were recorded it seems that the tide has really turned among philosophers of mind (Searle has his own biases but he tries to be impartial in explaining all sides). At least among philosophers who specialize in the subject it's hard to find many who seriously believe that a GPU running Resnet is conscious.
Obviously there's a whole semester's worth of material in those lectures and the arguments can't really be summed up in an HN comment, but anyone who is interested in thinking about these questions should really give the whole course a listen.
[1]: https://www.youtube.com/watch?v=zi7Va_4ekko