> So the question is, this simulated brain, which is behaving very similarly indeed to a biological brain: does it have consciousness? Does it feel?

If you answer "yes" to this question, what happens if you split the computations of that simulated brain over a million people equipped with pen and paper? Where is its consciousness happening now?

I have yet to hear a satisfying answer from anyone who believes that an AGI will automatically be conscious. But then again, I have yet to hear a satisfying explanation of consciousness from anyone...

Consider performing the algorithm of Alpha Go Zero through a million people equipped with pen and paper. One can hardly argue against this being possible in principle, but what is playing the game?

This is not proof that AGI will automatically be conscious, or even just that some sort of conscious machine intelligence is possible, it is merely intended to suggest that Searle's line of argument does not show its impossibility.

So I simplified the problem, by replacing the ability to understand Chinese with the ability to see in colour: http://www.fmjlang.co.uk/blog/ChineseRoom.html

While the simplified system is able to distinguish colours, how does it have the first person experience of seeing in colour?

So yes, the guy in your room can experience colors by using the sheets (though in a different and much less capable way than most of us, thus him not wanting to equate this with our common understanding of experiencing color), and is color blind without. And I kind of get deaf when plugging my ears, though that experience is vastly different from someone who never had hearing to begin with.

That doesn't answer what it is like "seeing red" and "feeling blue". I don't know what it's like for you. I can only look at your behavior, see it being similar to mine under the same circumstances and thus assume your internal state to be similar. Until your behavior tells me it being different... like someone blind running into a wall or someone approaching a chasm with no signs of fear.

ps: had to google "feel blue", as it meaning "depressed" isn't really a thing in my native language. Translating (color based) idioms is actually kind of telling... it seems only possible directly if both cultures had shared aspects / experiences.

So yeah, while conflating consciousness with intelligence doesn't make a lot of sense to me. But I'm not sure equating it with perception is getting us anywhere better (hope idiom didn't get lost in translation).

The person in my room (an achromat) can't experience colours at all, and they're able to tell you that. Their eyes have no functioning cones. They see everything in monochrome, so to them, watching a colour movie is how watching a black and white one is to us. Under low light conditions our cones are inactive and we see only in monochrome using our rods, just like achromats do. So people with normal colour vision can experience what it's like to be achromatic, but achromats can't even imagine what colours are like.

In the room, the person can only tell what colours the papers are by observing how dark they look when overlaid with the coloured filters (whose colours they know).

The metaphorical use of colours in the language isn't really important, and as you point out is dependent on culture.

There is a sense in which the system has an experience (necessarily first-'person'), but it is nothing like what we experience on seeing color. It is no more or less an experience than an ordinary color meter would have, and it lacks any awareness or concept of having an experience, no memory of it, no recognition of a similarity to previous experiences or of it being unlike any previous experience, no ability to imagine such experiences, no expectation of future experiences, and no emotional reaction to it ('my favorite color!') It is an experience only in the weakest sense possible.

I prefer a simpler analogy: what step in a sorting algorithm actually does the sorting? The question is actually nonsense, because every step is necessary, and removing any single step will break the sorting property for some inputs.

When you have all the relevant information, it's clear that the original question itself is ill-posed. We lack some of that information when it comes to.consciousness, hence why the Chinese Room is convincing to some.

You are right that we don't have a good (any?) definition of consciousness therefore it is difficult to properly ask if Chinese room has it. After reading and talking with people it seems that consciousness is something one can only claim to have (like a subjective feeling), but can't really manifest (I can only assume you are conscious because I believe I am, and you behave similarly enough).

But can a bigger system have consciousness when individual parts of it are already conscious? I don't see why this should necessary follow. I'm sure there are somewhat similar and interesting emergent properties, but we'd probably call it something different. Probably closer to the kind of group identities we already have with lots of humans interacting. Though obviously we'll know what it feels like, just like none of my neurons will ever feel like i do.

If you had a physics model it could describe it perfectly but it would likely involve a very large group of matricies and linear algebra constraints. You technically could work through every equation given a large set but examining the watch yourself would likely be quicker, if not learning to be a sufficiently good watchmaker.

However if you assemble a computer model to handle every component's interaction individuallt and a physics engine to do number crunching you could get a working mathematical model of the clock.

Bob: Of course.

Alice: But what if you scaled up all the neurons in the brain a factor of 2? Would it still be conscious?

Bob: Sure, if it's the same system running the same way. There's no reason to expect it to matter.

Alice: Then if it was a factor 4 larger?

Bob: Absolutely. Human brains are sized as they are as a matter of evolutionary coincidence. Bird neurons are much more compact than ours, and still manage much of the same work.

Alice: Right, but if every neuron also contained chloroplasts?

Bob: Which did what? There isn't much light there.

Alice: Nothing, they're completely ancillary.

Bob: Then yes, evidently it's still the same system... at what point does this become a hard question?

Alice: Just hold on a few orders of magnitude...

If you scale the system but keep one end tied down it isn't the same system.

[0] https://en.wikipedia.org/wiki/Pando_(tree)

My reason for comment related purely to the looseness of the conversation, w.r.t. the word "scale", which, it seems I had misunderstood to mean an affine transform.

In light of this comment, I now see you were talking about some growth function or change in topology/structure.

Thanks for the clarification.

The thing with information is that it is not as localized as other physical quantities like spin, mass, or even energy. Conversely if the information were localized we would call it lack of thereof. Information between systems mean correlation between them. Same goes with information processing where changes in one system corresponds to changes in the other. It happens in between.

Also, you run into the problem of time slices. One step in a simulation of my brain is not conscious. There must be some self-reference going on in there that give us this illusion over time that I doubt a pen and paper model could capture.

I'm of the school of thought that a turing machine, let alone pen and paper, might not even be capable of this simulation. For example, it's possible that there are quantum effects going on in biological neurons that touring machines are not equipped to simulate.

Of course, we can not yet rule out the possibility that there exist more powerful models of computation than Turing machines. We just know quantum effects don't lead there.

Our brain organization can be described by its various information cascades. But if conscious is implementing the right kind of information cascade, then a brain simulated by a million people would be conscious. The "where" of conscious is the same question as the "where" of informational relationships. But this isn't a well-formed question. There is no one location where some sequence of bits and its proper interpretation is stored. It's distributed among the components involved in the information cascade. The "where" of consciousness is the same.

So it's not surprising if we struggle to answer "Can machines be X?", when X is undefined.

Do you think beings in this second run of the universe are conscious? Or do you think that somehow something has been left out of the simulation?

So if there is conscious beings in the universe, it'll be in the simulation. You hold that you're conscious, so then I'd hope you'd agree the simulation has at least one conscious being in it.

I don't know why some folks consciousness test only seems to work for themselves. I've been convinced by the simple argument by induction: I am an example of an animal. I have consciousness, so others likely do too.

I haven't really found a compelling rational argument to fully reject solipsism.

Though I agree that looking for yourself in a perfect simulation is a clever hack. :-)

[1] https://books.google.com/books?id=7pFHBAAAQBAJ&lpg=PT51&ots=...

Here's Dennett on this very subject, about 20 years ago. In Consciousness Explained he imagines what a Turing Test conversation between a human 'judge' and a 'Chinese Room' might look like:

Judge: Did you hear about the Irishman who found a magic lamp? When he rubbed it a genie appeared and granted him three wishes. “I’ll have a pint of Guiness!” the Irishman replied and immediately it appeared. The Irishman eagerly set to sipping and then gulping, but the level of Guiness in the glass was always magically restored. After a while the genie became impatient. “Well, what about your second wish?” he asked. Replied the Irishman between gulps, “Oh well, I guess I’ll have another one of these.”

CHINESE ROOM: Very funny. No, I hadn’t heard it– but you know I find ethnic jokes in bad taste. I laughed in spite of myself, but really, I think you should find other topics for us to discuss.

J: Fair enough but I told you the joke because I want you to explain it to me.

CR: Boring! You should never explain jokes.

J: Nevertheless, this is my test question. Can you explain to me how and why the joke “works”?

CR: If you insist. You see, it depends on the assumption that the magically refilling glass will go on refilling forever, so the Irishman has all the stout he can ever drink. So he hardly has a reason for wanting a duplicate but he is so stupid (that’s the part I object to) or so besotted by the alcohol that he doesn’t recognize this, and so, unthinkingly endorsing his delight with his first wish come true, he asks for seconds. These background assumptions aren’t true, of course, but just part of the ambient lore of joke-telling, in which we suspend our disbelief in magic and so forth. By the way we could imagine a somewhat labored continuation in which the Irishman turned out to be “right” in his second wish after all, perhaps he’s planning to throw a big party and one glass won’t refill fast enough to satisfy all his thirsty guests (and it’s no use saving it up in advance– we all know how stale stout loses its taste). We tend not to think of such complications which is part of the explanation of why jokes work. Is that enough?

Dennett: “The fact is that any program that could actually hold up its end in the conversation depicted would have to be an extraordinary supple, sophisticated, and multilayered system, brimming with “world knowledge” and meta-knowledge and meta-meta-knowledge about its own responses, the likely responses of its interlocutor, and much, much more…. Maybe the billions of actions of all those highly structured parts produce genuine understanding in the system after all.”

His point (if I need to re-explain it?) is that the Chinese Room argument is superficially compelling. But if you actually follow through and imagine the Chinese Room doing the things that the thought experiment defines it to be doing, then you get a picture of an extraordinarily complicated system, and the nuts and bolts of exactly what that system is running on fall away into irrelevance.

I think this is a common failing of thought experiments. Gedankenexperiments help us ask better questions and design better real experiments, but if you're trying to use them to make fundamental conclusions about reality in an a priori way, you're likely going to fail.

You can grow biological brain structures from stem cell. It's called Brain Organoids and Neural Organoids, that may one day be used in machines and may one day have consciousness properties. Interview about this with Lex Fridman (MIT Artificial Intelligence teacher) and Paola Arlotta (Harvard brain neuroscientist): https://lexfridman.com/paola-arlotta/

> We don’t think of a few cells as having consciousness, but somehow the broader collection does.

This is called "emergent property" (see "Emergence" in Wikipedia), that is a sub-area of Complex Systems theory. There's a theory that states it may be possible that consciousness may appear as an emergent property from a high complex, dynamical and evolutionary AI system.

> imagine an incredibly-detailed computer simulation

Some paradigms to approach AGI (Artificial General Intelligence / Superintelligence) are: 1. Simulate the brain in low level details; 2. Simulate the functioning of higher level structures of the brain (cortical columns, visual cortex, hippocampus...); 3. Create artificial neurons (that can be used to neural networks, deep learning, etc.); 4. Simulating simple brains and nervous systems (e.g jellyfish, insects, C.elegans connectome project, etc) to understand how intelligence works; 5. Simulating the behavior of the brain ("cognitive architectures"); 6. Creating en evolutionary system that could converge to brain-like funcionality.

It probably helps to think about some edge cases to get some insight.

What happens when you sleep and you loose sense of conciousness?

What happens in cases of brain injury where concousiness goes into weird failure modes - thinking it wakes up on the same state every day, thinking it's -1month or so, collapsing etc.

What happens in cases of distorted/malfunctioning brain schizophrenia etc.

I've read somewhere that if you put implant which mimics neuron activity, your brain will build structure around it just like with normal neurons. What would happen if you take 1 square mm of brain and replace it with implant, slowly replacing the whole brain? What happens when you attach brain to larger structure/orders of magnitude more complex one, will conciousness transfer there? Can you disconnect original brain leaving conciousness?

As we don't have powerfull computers yet to compete with brain, is it possible to connect two brains instead?

Memory is likely playing central role. You are what you remember it seems. Can you boost memory capacity with some implants, will it make more-conciousness?

There are many not yet feasible experiments, but some should be. Also completely brutal in many cases.

Look up split brain experiments.

I wonder if it's possible to quantify concousness and how much of it you loose it. It seems like it's not half/half split, ie. 1+1 != 2; can you split them again and how far you can do it.

I also wonder what would happen if you left them to be for a while and then connect them back, would they merge or stay diverged.

Maybe you could run it on ordinary neural net, it would be easier to quantify. Maybe some numbers would reveal something interesting, ie. maybe neural nets are exponentially better when connected and that's the whole trick. It won't answer questions about counciousness directly though.

So no, I don't think it's possible to create a simulated brain that behaves similarly as a human, as the biological medium is in a constant state of dissolution, repair, evolution, and adaptation. This is how we learn, get stressed, get tired, get certain emotions...it's a whole new layer of programming you'd need to perfect beyond just replicating a brain at one moment in time.

A brain in silico would be apples:oranges different than a biological brain.

Maybe I'm making a massively overly pedantic point. It's just that your post began with a claim that certain things "cannot be given" to an artificial mind, which sounded like you were proposing absolute limits to AI.

My guess? No, it won’t be conscious or even meaningful.

More fundamentally, our inability to answer this question drives at our ignorance of our own neurology. We don’t know the building blocks of thought. It might happen at the neurochemical or electrochemical level. It may even depend on quantum effects.

[0] https://duckduckgo.com/?q=glia+cells+NMDA&atb=v173-1&ia=web

What about "swapping" one-by-one vs all-at-once? Would that matter?

What I'm getting at - which is where things seem to go sideways to think about it - is that if you did this swap - but rather the artificial neurons remain outside the body, and you did it near-instantly (vs slowly one-by-one) - and I am speaking of relative speeds here; say "near-instantly" means the entire swap takes one second, vs the slower manner of each swap taking a millisecond (or whatever).

Would that matter?

Now - what if instead of swapping - you did a parallel simulation instead - one-by-one, recreating the brain, but the artificial version operating in lockstep with the original; when one neuron "fires" the same artificial neuron "fires", same "paths" taken, etc.

Then you kill (choose your method and make it quick) the natural brain - instant swap? Or is there something different? Where does the "consciousness" go? If it is different, why is it any different than the "near-instant" swap?

Why would making that "lockstep-copy" matter, vs not making a "copy"?

I think you get what I am saying. Think on it a bit. There isn't a good answer that I am aware of.

Note: I'm assuming an "instant kill" - death to the natural brain faster than neural signals can travel neuron-to-neuron, ideally. We can posit the idea that if it were any slower (and especially if it were really slow) that the two brains would diverge in experiences, and would become two different "consciousnesses". But it does make you wonder why this should happen with a copy, vs not (in theory?) with a one-by-one swap. Heck - maybe there's an answer in there somewhere...

This seems to be a non-sequitur. If we simply understand consciousness as a property of the brain, as long as you have 2 copies of the same brain, you naturally have 2 consciousnesses. The 2 may be equal or not (most likely not, given that random noise is extremely likely to play a role in brain processes), but they are definitely not a single object.

Supposing that the mind is the result of neurons' inherent consciousness properties is like attributing morphine's effect as arising from its "dormitive virtue" - it does not advance our understanding.

Also, it seems quite plausible that an aggregation of semiconductor junctions could have the same computational abilities as a neuron. This is, in fact, an active area of R&D.

Why not? Digital stuff is made up of matter as well. Unless there's something special about neurons when it comes to consciousness, i.e. the conscious bits reside in neurons and nowhere else for some reason, there's reason to think non-biological structures that produce the same output would also be conscious. Any substantive theory of consciousness, materialist or not, will need to include a place for structure and dynamics, i.e. information cascades, within their theory to account for the correlations we observe between conscious states and brain structure and dynamics. But these qualities are present in a digital implementation as well.

I'm not sure what could possibly answer this problem, at least to the satisfaction of the people who think it is a problem. Personally, I'm relatively happy that Daniel Dennett's ideas demonstrate how what we think of as consciousness, qualia etc can be explained via mechanistic/algorithmic processes.

They will also go on internet forums arguing that while they, of course, do have consciousness - the digital replicas of themselves would not!

We don't experience minutiae of activations of billions of neurons, or the work of glia. We experience "executive summaries" of them. So it's natural to assume that those summaries can be represented in another physical form.

Why physical representation of those "summaries" are there? Because it's a way our brains do high level decision making.

This leaves out the question why we experience anything at all. And I have trouble imagining that there are physical processes (quantum or not), which create a mysterious consumer for "executive summaries". If it was so, we would have decision-making pathway and useless subjective experience pathway. It's hard to explain from evolutionary standpoint.

I prefer to think that "executive summaries" abstracted from physical representation just exist without any need for mysterious consumers. Not unlike how natural numbers exist according to mathematical platonists.

Is it logically possible for a universe that is physically identical to ours, but where physical representations of natural number 2 correspond to natural number 3 to exist? Or where physical representations don't correspond to any natural numbers?

Another funny take on inverted qualia argument. We can subitize small numbers of items, that is feel the number of items without counting. Can you imagine someone identical to you, who feels the number of two apples as you feel the number of three apples?

I think there are more easily accessible examples that investigate this problem closer. Are animals conscious? How about children? How about very slow people?

But of course it's completely unknowable. Maybe every other single creature on earth is just an unfeeling automaton, and only I feel.

But I doubt it. ;-)

If we can, then indeed consciousness comes from the collective function of individual cells. But maybe there's more to the workings of the brain?

Joking aside, I agree with you. My point was simply that we're very ignorant about the brain and its workings, one way or another.

The laws of physics start to become relevant here. It is quite likely that it is not possible to simulate a human brain using a classical computer, at least not at any kind of speed such that it can experience the world as we do.

Eventually the brain is entirely silicon, but to the owner subjectively nothing has changed, they are as 'conscious' as they ever were.

I worry we'll never know the answer to this, even if we manage to pull off such a simulation.

I would never make a copy of my brain (or somebody else's) with the intention of disconnecting it later.

Not really.

> So the question is, this simulated brain, which is behaving very similarly indeed to a biological brain: does it have consciousness?

Obviously yes. Either that or our brains are filled with magic unicorn dust.

On the other hand, if there is some guiding (like a programmer), the artificial brain can be functional. If the real brain would be guided by some kind of consciousness, this would work the same.

The consciousness needs to be in a different dimension, in a way that physics does not change. So it would be like a 2D pool table, where we can describe exactly the movement of every ball. But sometimes a ball gets hit by a player, outside the table. And this gives us events that bring the balls to a certain direction.

In similar sense the brain can be played by a consciousness to get it working. In a way that is very hard to detect with physical instruments.

Okay, we have a proof that a machine can't solve the halting problem for every arbitrary Turing machine, but we can build algorithms that solve it for some cases.

Now, can a human solve the halting problem for every arbitrary Turing machine? We obviously can't, either. Make the machine complex enough and we can't wrap our minds around it.