Consider performing the algorithm of Alpha Go Zero through a million people equipped with pen and paper. One can hardly argue against this being possible in principle, but what is playing the game?

This is not proof that AGI will automatically be conscious, or even just that some sort of conscious machine intelligence is possible, it is merely intended to suggest that Searle's line of argument does not show its impossibility.

So I simplified the problem, by replacing the ability to understand Chinese with the ability to see in colour: http://www.fmjlang.co.uk/blog/ChineseRoom.html

While the simplified system is able to distinguish colours, how does it have the first person experience of seeing in colour?

So yes, the guy in your room can experience colors by using the sheets (though in a different and much less capable way than most of us, thus him not wanting to equate this with our common understanding of experiencing color), and is color blind without. And I kind of get deaf when plugging my ears, though that experience is vastly different from someone who never had hearing to begin with.

That doesn't answer what it is like "seeing red" and "feeling blue". I don't know what it's like for you. I can only look at your behavior, see it being similar to mine under the same circumstances and thus assume your internal state to be similar. Until your behavior tells me it being different... like someone blind running into a wall or someone approaching a chasm with no signs of fear.

ps: had to google "feel blue", as it meaning "depressed" isn't really a thing in my native language. Translating (color based) idioms is actually kind of telling... it seems only possible directly if both cultures had shared aspects / experiences.

So yeah, while conflating consciousness with intelligence doesn't make a lot of sense to me. But I'm not sure equating it with perception is getting us anywhere better (hope idiom didn't get lost in translation).

The person in my room (an achromat) can't experience colours at all, and they're able to tell you that. Their eyes have no functioning cones. They see everything in monochrome, so to them, watching a colour movie is how watching a black and white one is to us. Under low light conditions our cones are inactive and we see only in monochrome using our rods, just like achromats do. So people with normal colour vision can experience what it's like to be achromatic, but achromats can't even imagine what colours are like.

In the room, the person can only tell what colours the papers are by observing how dark they look when overlaid with the coloured filters (whose colours they know).

The metaphorical use of colours in the language isn't really important, and as you point out is dependent on culture.

There is a sense in which the system has an experience (necessarily first-'person'), but it is nothing like what we experience on seeing color. It is no more or less an experience than an ordinary color meter would have, and it lacks any awareness or concept of having an experience, no memory of it, no recognition of a similarity to previous experiences or of it being unlike any previous experience, no ability to imagine such experiences, no expectation of future experiences, and no emotional reaction to it ('my favorite color!') It is an experience only in the weakest sense possible.

I prefer a simpler analogy: what step in a sorting algorithm actually does the sorting? The question is actually nonsense, because every step is necessary, and removing any single step will break the sorting property for some inputs.

When you have all the relevant information, it's clear that the original question itself is ill-posed. We lack some of that information when it comes to.consciousness, hence why the Chinese Room is convincing to some.

You are right that we don't have a good (any?) definition of consciousness therefore it is difficult to properly ask if Chinese room has it. After reading and talking with people it seems that consciousness is something one can only claim to have (like a subjective feeling), but can't really manifest (I can only assume you are conscious because I believe I am, and you behave similarly enough).

But can a bigger system have consciousness when individual parts of it are already conscious? I don't see why this should necessary follow. I'm sure there are somewhat similar and interesting emergent properties, but we'd probably call it something different. Probably closer to the kind of group identities we already have with lots of humans interacting. Though obviously we'll know what it feels like, just like none of my neurons will ever feel like i do.

If you had a physics model it could describe it perfectly but it would likely involve a very large group of matricies and linear algebra constraints. You technically could work through every equation given a large set but examining the watch yourself would likely be quicker, if not learning to be a sufficiently good watchmaker.

However if you assemble a computer model to handle every component's interaction individuallt and a physics engine to do number crunching you could get a working mathematical model of the clock.

Bob: Of course.

Alice: But what if you scaled up all the neurons in the brain a factor of 2? Would it still be conscious?

Bob: Sure, if it's the same system running the same way. There's no reason to expect it to matter.

Alice: Then if it was a factor 4 larger?

Bob: Absolutely. Human brains are sized as they are as a matter of evolutionary coincidence. Bird neurons are much more compact than ours, and still manage much of the same work.

Alice: Right, but if every neuron also contained chloroplasts?

Bob: Which did what? There isn't much light there.

Alice: Nothing, they're completely ancillary.

Bob: Then yes, evidently it's still the same system... at what point does this become a hard question?

Alice: Just hold on a few orders of magnitude...

If you scale the system but keep one end tied down it isn't the same system.

[0] https://en.wikipedia.org/wiki/Pando_(tree)

My reason for comment related purely to the looseness of the conversation, w.r.t. the word "scale", which, it seems I had misunderstood to mean an affine transform.

In light of this comment, I now see you were talking about some growth function or change in topology/structure.

Thanks for the clarification.

The thing with information is that it is not as localized as other physical quantities like spin, mass, or even energy. Conversely if the information were localized we would call it lack of thereof. Information between systems mean correlation between them. Same goes with information processing where changes in one system corresponds to changes in the other. It happens in between.

Also, you run into the problem of time slices. One step in a simulation of my brain is not conscious. There must be some self-reference going on in there that give us this illusion over time that I doubt a pen and paper model could capture.

I'm of the school of thought that a turing machine, let alone pen and paper, might not even be capable of this simulation. For example, it's possible that there are quantum effects going on in biological neurons that touring machines are not equipped to simulate.

Of course, we can not yet rule out the possibility that there exist more powerful models of computation than Turing machines. We just know quantum effects don't lead there.

Our brain organization can be described by its various information cascades. But if conscious is implementing the right kind of information cascade, then a brain simulated by a million people would be conscious. The "where" of conscious is the same question as the "where" of informational relationships. But this isn't a well-formed question. There is no one location where some sequence of bits and its proper interpretation is stored. It's distributed among the components involved in the information cascade. The "where" of consciousness is the same.

So it's not surprising if we struggle to answer "Can machines be X?", when X is undefined.

Do you think beings in this second run of the universe are conscious? Or do you think that somehow something has been left out of the simulation?

So if there is conscious beings in the universe, it'll be in the simulation. You hold that you're conscious, so then I'd hope you'd agree the simulation has at least one conscious being in it.

I don't know why some folks consciousness test only seems to work for themselves. I've been convinced by the simple argument by induction: I am an example of an animal. I have consciousness, so others likely do too.

I haven't really found a compelling rational argument to fully reject solipsism.

Though I agree that looking for yourself in a perfect simulation is a clever hack. :-)

[1] https://books.google.com/books?id=7pFHBAAAQBAJ&lpg=PT51&ots=...

Here's Dennett on this very subject, about 20 years ago. In Consciousness Explained he imagines what a Turing Test conversation between a human 'judge' and a 'Chinese Room' might look like:

Judge: Did you hear about the Irishman who found a magic lamp? When he rubbed it a genie appeared and granted him three wishes. “I’ll have a pint of Guiness!” the Irishman replied and immediately it appeared. The Irishman eagerly set to sipping and then gulping, but the level of Guiness in the glass was always magically restored. After a while the genie became impatient. “Well, what about your second wish?” he asked. Replied the Irishman between gulps, “Oh well, I guess I’ll have another one of these.”

CHINESE ROOM: Very funny. No, I hadn’t heard it– but you know I find ethnic jokes in bad taste. I laughed in spite of myself, but really, I think you should find other topics for us to discuss.

J: Fair enough but I told you the joke because I want you to explain it to me.

CR: Boring! You should never explain jokes.

J: Nevertheless, this is my test question. Can you explain to me how and why the joke “works”?

CR: If you insist. You see, it depends on the assumption that the magically refilling glass will go on refilling forever, so the Irishman has all the stout he can ever drink. So he hardly has a reason for wanting a duplicate but he is so stupid (that’s the part I object to) or so besotted by the alcohol that he doesn’t recognize this, and so, unthinkingly endorsing his delight with his first wish come true, he asks for seconds. These background assumptions aren’t true, of course, but just part of the ambient lore of joke-telling, in which we suspend our disbelief in magic and so forth. By the way we could imagine a somewhat labored continuation in which the Irishman turned out to be “right” in his second wish after all, perhaps he’s planning to throw a big party and one glass won’t refill fast enough to satisfy all his thirsty guests (and it’s no use saving it up in advance– we all know how stale stout loses its taste). We tend not to think of such complications which is part of the explanation of why jokes work. Is that enough?

Dennett: “The fact is that any program that could actually hold up its end in the conversation depicted would have to be an extraordinary supple, sophisticated, and multilayered system, brimming with “world knowledge” and meta-knowledge and meta-meta-knowledge about its own responses, the likely responses of its interlocutor, and much, much more…. Maybe the billions of actions of all those highly structured parts produce genuine understanding in the system after all.”

His point (if I need to re-explain it?) is that the Chinese Room argument is superficially compelling. But if you actually follow through and imagine the Chinese Room doing the things that the thought experiment defines it to be doing, then you get a picture of an extraordinarily complicated system, and the nuts and bolts of exactly what that system is running on fall away into irrelevance.

I think this is a common failing of thought experiments. Gedankenexperiments help us ask better questions and design better real experiments, but if you're trying to use them to make fundamental conclusions about reality in an a priori way, you're likely going to fail.