It hides the conscious understanding, presumably necessary to pass a Turing test, within the "program".

The human in the story is merely a decoy performing robotic tasks.

Let's suppose we build the Cosine room. The room is full of 10 year olds that haven't yet taken trig. Each performs the function of a transistor, and has no idea of what they're doing, other than blindly executing the instructions they were given. None of the participants has the slightest clue about what a cosine is. Yet, the output of the room will still be cos(x). Thus I think it's fair to say that the room as a whole implements cos(x).

In the same way, I don't see why wouldn't we conclude the room is speaking Chinese. It doesn't matter how it manages to do so and what it's made of. If it quacks like a Chinese duck, then it's a Chinese duck.

And yet some people seem to understand Chinese just fine. How can you explain that gap? It's only a stupid argument if you first assume that functionalism is true.

The argument is supposed/or often used to disqualify machines from consciousness but it's stupid because biological neurons don't understand the bigger picture anymore than an artificial neuron sampled from a model.

If you show a 2 years old kid, an apple and you say "This is an apple", now the kid knows the thing - the apple - is an apple in his world model. Automatically inherits LOTs of properties, you can ask the kid and see in real time how he began inmediately to associate the physical object - now named in its internal language model - to some other similar stuff he alredy knows, like "This is plant?", "It falls from a tree like an orange", "It has some skin like an orange", "can you cook apples like you cook bananas?", an so on.

But this requires a physical representation of the apple, now kids intelligence has some edge here, it can do the same thing just with words, you can teach them words, "apple", and say them "it's a fruit", if they have another fruit already "tagged", like a banana, they will say you almost inmediately, "is it tasty like bananas" ("tasty" is code for sweet in child's language models around the planet).

Hence, the LLM could have an emergent property of actually knowing what every word the "say" mean, if - like many have been inferring lastly - they also have a world model, it would relatively easy to just "plug", let's say 10 million of words to their exact meaning, and even to their relative meaning depending on the context they're being used.

And that's precisely what we may be seeing right now when we prompt something to chatGPT, and all the mathematic stuff, like "predicting the next word" is just some really, really low level process inside the LLM, not much different than the electric stuff - watchable by EEG - happening between neurons in the brain.

So if you look at the "EEG" from a LLM, the prediction thing happening inside the LLM, it won't probably tell you much, just like having a casual look to an EEG won't tell you much about what the person was thinking at the time of the capture of the EEG.

The second quote doesn't illuminate that. Which biological processes break the symmetry?

On the first hand, all communication is purely syntactic; letters, words, gestures, and every other formulation of communication is the transfer of symbols, which cannot carry semantic information. They can point to it, they can imply it, but you'll have to dance pretty hard to actually prove the transfer of semantic information. If you and I both have some semantic concept associated with the symbol "China" in our mental landscapes, neither I nor you, nor anyone, has any way determining whether that is the same semantic concept. In fact, it almost assuredly isn't, since those semantic concepts would have to be built from our unique personal histories, which are unique.

On the second, the fundamental assumption of semantic communication is that I feel like I have semantic shenanigans going on in my mental life and Searle sort of looks like me, so I should assume that he has semantic stuff going on. Not only is that not a particularly well-founded assumption, I can't even be sure I have semantic contents in my mental life. Introspection isn't exactly reliable, right?

On the third hand, you have a choice: if you do not agree with Penrose, et al, on the quantum nature of the magic of mind, then Searle's "biologically specific powers" are chemistry. Now, I never actually took a chemistry class, but I don't believe there is any chemical reaction that comes with a label of semantics. It's all physics anyway, which I know cannot introduce semantics.

If you do agree with Penrose (which I understand Searle does), the situation is not actually better. Quantum mechanical antics are random, right? Bell's Inequality and all? Being fundamentally, quantum mechanically, random is not in any sense better from a semantic standpoint than being syntactically deterministic.

So my bottom line is that I don't know if I have semantic contents, I absolutely don't know if Searle has semantic contents (and I'm not feeling generous), and I do not know of any way that he could prove to me that he does have "semantic contents". So, yeah.

You notice I only addressed the second paragraph from Searle. That is because the fundamental flaw in his Chinese Room "demonstration" is on obvious display in his final sentence, "The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese"; it is purely, straight up, dualistic. He is asserting that the person in the room has some magic property of "consciousness or intentionality" and that if the person in the room does not have the "semantic capability" of understanding Chinese, then it is impossible despite all evidence that anything in the scenario has the semantic capability of understanding Chinese.

You could put a native Chinese speaker in the room, reading and responding to the inputs, and he claims it would have "consciousness or intentionality"; you then replace her with me (who didn't exactly dazzle at Spanish and Latin) and even if I produce the same outputs in response to the inputs as she does, there is no "consciousness or intentionality". The difference is simply a magic, intangible, imperceptible soul.

Whoo. Went off on that one.

All formal communication is syntactic, but not all communication is formal, and not all experiences are acquired via communication.

Take the word “pizza”. If you’ve ever seen, smelled, touched, or tasted pizza, then you’ve acquired semantic information about pizza that could not possibly be acquired by syntax alone.

Same goes for informal communication such as eye contact, touch, hugs, kisses, handshakes, sex, etc. These are experiences people share that cannot be acquired through any other means.

Why does this matter for the AI debate? Suppose I ask a master pizza chef for tips to improve my pizza recipe. This chef has presumably many years of pizza-making and tasting experience. The chef can make recommendations to improve the flavour of the pizzas I make at home that are informed by semantic contents; actual knowledge of what effect the ingredients and methods have on the taste.

An AI can do none of that. It can only take the written words (syntactic inputs) of many pizza chefs and synthesize a model for pizza recipes. From that it can send me a bunch of words which, when followed, may or may not improve my pizza recipe.

It has no knowledge whatsoever, not least of which is experience of making and eating pizza. It cannot tell a good recipe from a bad one, unless its training set was specifically labeled to do that. Furthermore, models in general are highly vulnerable to adversarial inputs which would make it trivial for someone to construct a bad pizza recipe that gets misclassified as a good one.

Lastly, the model would be no help at all if I were to suggest a novel topping to add to the pizza. Whereas the pizza chef, if they had the experience of tasting that topping separately from pizza, would be able to imagine how it might taste on a pizza and make an informed recommendation. Furthermore, the pizza chef might even try that novel topping on a pizza and get back to me, something a language model could never do.

And this was Searle’s original argument. You can’t build intelligence by building strictly symbolic machines (computers). You have to endow them with semantic content by some other means.

I will note that your semantic conception of pizza is likely very different from mine, or from my friend's, who doesn't like cheese. You may have a semantic conception of pizza, but I have no way of knowing that.