This is called the systems response to the Chinese Room Argument. [0]

Searle's response [0] goes something like this:

You're missing the point. Suppose the man in the room memorised the program, so that he could answer the questions himself with no need of external aids. Now, the man and the system are the same, and yet by your account, the man doesn't speak Chinese, but the system does.

Personally I find the whole argument to bear no clear connection to the questions of consciousness. The question of whether a system consciously understands a problem is muddled, as it isn't clear a priori that problem-solving competence has any connection to consciousness. To use Dennett's term, a system can be competent without comprehension. A pocket calculator is quite unable to explain what it's doing, but is able to perform superhuman number-crunching. The argument may succeed in tying the reader in a knot, but I figure that's just because most people haven't thought much about what conscious comprehension really means.

Another problem with the Chinese Room Argument is that, if it really holds up, it ought to hold up just as well against the human species itself. If you're going to attack the idea that consciousness can arise from non-conscious components, where does that leave us?

[0] https://en.wikipedia.org/wiki/Chinese_room#Systems_and_virtu...

It fails to demonstrate anything interesting about consciousness. It certainly doesn't demonstrate that computer systems can never be conscious in the way we can. Nothing in the argument applies any more or less to neurons than to transistors.

If a person is manually executing an algorithm (the memorised Chinese Room algorithm, or naive matrix multiplication, or whatever), then we could distinguish between the algorithm and the person executing it. In that sense there are two computation streams at work. That doesn't show that a second consciousness has been brought into being, though. I wouldn't conclude that there are two 'minds'.

Your comparison to virtual machines is a good one, it's one that Dennett uses.

If the executed algorithm is consciousness, then a second consciousness has been brought into being, same as for first consciousness.

A lot hinges on If the executed algorithm is consciousness.

I can see the sense in that argument though, in that the 'first consciousness' is acting as the computational substrate for the 'second consciousness', the way the physical action of neurons acts as our computational substrate. I'm not convinced this maps to using a card system to have a conversation in a foreign language. It doesn't seem self-evident that doing so should be considered enough to demonstrate consciousness, it's only enough to demonstrate than the algorithm is effective in having a conversation.

It's not self-evidently the equivalent of an accurate real-time computer simulation of a specific person, for instance. I think you can make a strong case that such a system would amount to a conscious person and should be treated as such. (It would follow from this that shutting down a holodeck simulation of someone is morally fraught.) The only alternative would be to morally privilege neuron-based architectures over transistor-based architectures, which seems like an uphill philosophical battle.

I don't put much stock in the question of whether the card-based algorithm 'understands' the conversation it is having. Is it competent? Yes. Is it conscious? I'm not convinced it is. Asking whether it understands is to conflate these two questions.

>It's not self-evidently the equivalent of an accurate real-time computer simulation of a specific person, for instance. I think you can make a strong case that such a system would amount to a conscious person and should be treated as such.

This was touched recently in art. If you're interested, it's "Sword art online Alicization". The shutdown problem is applicable to AI too. What's new is that the life of virtual people is shown at length and discussed questions as to what identity those people should have, what worldview, religion, philosophy, pride, dignity, justice.

I'm not sure that's the case. Simple chat programs can do a pretty good job simulating a human interlocutor. For a 'full' simulation, which we need by definition, we'd need something much more sophisticated (able to reason about all sorts of abstract and concrete things), but conceivably the solution might be very different from brain-simulation.

The thought experiment can easily be adjusted to close the door on my objection here: rather than a man in a room with an enormous card index, we have a pretty accurate real-time computer simulation of some specific person. That way the computational problem is defined to be equivalent to something we consider conscious. Laboriously computing that simulation by hand (presumably not in real time but instead over millennia) would change the substrate, but not the computational problem. If we're ok with there being an outer host consciousness and an inner hosted consciousness, the thought experiment poses no problem.

Of course, this isn't the position I started at, but it makes some sense that the real meaning of the thought experiment change as we adjust the simulated process. If our man were looking up the best moves to play tic-tac-toe, it would be plainly obvious that we're looking at competence without comprehension. If he's instead simulating the full workings of a human brain, the situation is different. The foreign language problem is somewhere between these extremes.

> It's a good part about artificial algorithms that they are transparent and we can show that they aren't only effective in conversation, but have everything to be had behind that conversation. It's due to the latter. Simply put, the algorithm isn't GPT, but AGI.

I'm not sure I quite follow you here. I agree that the depth and detail of the simulation is an important factor.

> What's new is that the life of virtual people is shown at length and discussed questions as to what identity those people should have, what worldview, religion, philosophy, pride, dignity, justice.

In contrast to Ex Machina that had the computer as a sociopathic villain with only surface level feigning of normal human emotion and motivation.

While we're vaguely on the topic, homomorphic crypto also puts a spin on things. We know it's possible for a host computer to be entirely 'unaware' of what's going on in the VM that it's running, in a cryptographic sense. Related to this, I've long thought that there's a sticky 'interpretation problem' with consciousness (perhaps philosophers have another term for it) that people rarely talk about.

If you run a brain simulator inside a homomorphically encrypted system, such that no one else will ever know what you're running in there, does that impact whether we treat it as conscious? Part of it is that the simulated brain isn't hooked up to any real-world sensors or actuators, but that's just like any old brain in a jar. Philosophically pedestrian. This goes far beyond that. Someone could inspect the physical computer, and they'd have no idea what was really running on it. They'd just see a pseudorandom stream of states. If there's consciousness inside the VM, it's only there with respect to the homomorphic crypto key!

If we allow that to count as consciousness, we've opened the door to all sorts of computations counting as consciousness, if only we knew the key. We can take this further: we can always invent a correspondence such that any sequence of states maps to a computation stream that we would identify as yielding consciousness. This looks like some kind of absurd endgame of panpsychism, but here we are.

Is there an alternative? I'm increasingly of the opinion that it seems like a non-starter to try to deny that transistor-based computers could ever be the substrate of consciousness. Short of that, where else is there to go?

</sidetrack>

But the entropy of the encryption key is the degree to which the consciousness is "hidden". This entropy is still massively lower than the entropy of the matter that makes up a brain. If we have some way to demonstrate the encrypted system is computing the mind program, say, by interacting with its input/output, then we can in theory demonstrate the system is conscious. The fact that the encrypted system's operation maps to a mind program with entropy equal to the key, rather than equal to the entropy of the bits in the mind program entails that the encrypted system intrinsically encodes the mind program. If the mapping were equivalent to just mapping the states of an arbitrary system to the mind program, the entropy would be equal to the much greater bits in the program. Comparing the entropy is the key differentiator.

Seems fair.

> But the entropy of the encryption key is the degree to which the consciousness is "hidden". This entropy is still massively lower than the entropy of the matter that makes up a brain.

Sure, there are far more possible states for a brain, than possible 4096-bit keys (for example).

> If we have some way to demonstrate the encrypted system is computing the mind program, say, by interacting with its input/output, then we can in theory demonstrate the system is conscious.

Right, although if we further adjust the thought experiment we run into a sort of 'systems argument' problem.

Suppose the homomorphically encrypted system uses an encrypted channel to communicate with actuators, such that a decrypter module is needed to connect it up. (We need not encrypt the channels from the sensors.) In that case, the homomorphically encrypted brain simulator plus the decrypter module, adds up to what we call a conscious system. On its own, the homomorphically encrypted brain simulator does nothing of interest, or at least, appears to do nothing of interest.

> The fact that the encrypted system's operation maps to a mind program with entropy equal to the key, rather than equal to the entropy of the bits in the mind program entails that the encrypted system intrinsically encodes the mind program.

That's what we'd typically expect of a cryptographic system, but now we have a sliding scale. What if the key were so large that it outweighed the state-space of the machine itself? Do we conclude that the length of the key determines how conscious the system is?

We could say that the contribution of homomorphic crypto is just that it permits us to use vastly smaller (pardon the oxymoron) keys to scramble states and their progressions.

What if we define another cryptographic scheme such that the correspondence formerly represented by a very long key, is instead represented by just a few bits? (Finally a way to tie philosophy of mind to Kolmogorov complexity!) Or perhaps I'm misunderstanding the point about entropy here?

But the entropy considerations are just a practical way for us to identify computational systems with certainty. It isn't an identity criteria. Consider your homomorphically encrypted program where the key is longer than the state space for that program. Presumably we cannot tell this encrypted program apart from some random set of operations that computes nothing (I doubt this is true in practice, but lets go with it). How can we say this program is in fact computing something? The assumption that the program is homomorphically encrypted also says there is a magic string of bits that unlocks its activity. Further, this magic string is independent (i.e. has zero mutual information) with the program in question. Essentially the key is random and so it cannot provide any information about the program itself. So when the key is combined with the encryption scheme to produce the decrypted program, we know that the program was embedded in the encrypted system the whole time, not added by the application of the key.

The key point is that information doesn't just pop into existence, information requires upfront payment in (computational) entropy, a computation-like process that does something like guess-and-check over the state space of the information. If ever you have a string of information, you either got it from somewhere else or you did guess-and-check over the state space. In the case of the homomorphic encryption, we know the key is independent of the program and so the key does not secretly contain the program. Thus the program must already exist in the behavior of the encrypted system.

We don't know the key but we know it exists by assumption. "Exists" here just means the upfront computational cost has already been paid for the relation between the hidden program, the encrypted system and the decryption key. Indeed, we can in theory recover the encrypted program with comparatively zero computational work by using the key. This is in contrast to recovering the mind program from, say, the dynamics of my wall. No upfront computational cost has been paid and so I have to search the entire state space to find a mapping between the wall and the mind program. Thus the wall provides no information about the mind program, i.e. it is not computing the mind program.

>The Chinese room argument certainly only exists to muddy the waters.

Does not compute. :)

The Chinese room argument, as I understand it, is not about whether the human understands Chinese. The question is whether the room understands Chinese.

Personally I place the rejection of functionalism on basically the same level as the acceptance of dualism. I don't know what precisely has happened to make most serious researchers reject dualism, but perhaps non-functionalism will run the same course.

I really don't understand why people have such a hard time with this argument. Seems like there's some unaddressed materialist/eliminativist prejudice underneath it all.

The program isn't doing anything because it's just a bunch of symbols. You need something to "animate" the program, but that something need not comprehend the symbols it is given (hence the whole idea of an effective method). (Actually, Searle goes further when he describes computation as observer relative, and I would extend this to programs. That is, there is not an objective fact of the matter that a computer is a computer and that it is computing. If you looked at it, there's nothing in what's going on that says "oh, yeah, this machine is adding numbers". Kripke also gets into this with his "quaddition" argument.)

When you understand Chinese, reading a symbol or legal string of symbols leads your mind to form conceptual content with some intentionality or signification. We can have other thoughts in response and this can lead to the production of signs with some other signification. The program lacks this semantic component and merely performs what amounts to a syntactic translation of the input signs into output signs. This process is entirely stripped of any semantic element. And that's the point. Computers are highly systematized patterns of convention that permit the simulation of some of what we would typically expect of human beings or maybe some other animal, but strictly speaking, they don't even compute anything, much less understand.

Also your assumption of the program not forming any conceptual content, will depend on what exactly the program is doing. The Chinese room says it's carrying out a conversation in a manner indistinguishable from a human, but that could mean anything from running a very fancy GPT-3 to running a whole-brain simulation of a Chinese person.

In either case, the human running the program is a distraction. You might as well ask if the pencil or the book he's using to compute the instructions understands Chinese. If the program was running on a CPU, it'd be the same thing - the CPU doesn't understand Chinese, the program does.

But the symbols aren't the interesting piece of a sequence of computation, but rather the structure being instantiated. There is a map/territory ambiguity here. The symbols are just placeholders into an abstract structure that describes the dynamics of some system. When the structure is reified, i.e. made actual by being implemented in a system, then it is the structure that has causal efficacy in the world and it is that which we interact with. Dismissing the idea of an implemented program understanding Chinese because "the program doesn't understand the symbols" is confusing the map for the territory.

>Searle goes further when he describes computation as observer relative, and I would extend this to programs.

I totally disagree. If I have a program that inverts an arbitrary matrix, this is not a function I project onto the program. This function is intrinsic to the sequence of operations it carries out. An intelligent alien studying the operation of the program would be able to figure out its function given enough time and effort.

>The program lacks this semantic component and merely performs what amounts to a syntactic translation of the input signs into output signs. This process is entirely stripped of any semantic element.

The CPU is performing syntactic translation, but the structure being implemented is an existing thing beyond CPU operations. And this is the misdirection in the Chinese room argument. To ask if the man understands Chinese is just to ask the wrong question.

You mean you're an eliminativist?