Yes, it has shown that we might progress towards AGI without ever having anything that is sentient. It could be nearly imperceptible difference externally.

Nonetheless, it brings forward a couple of other issues. We might never know if we have achieved sentience or just the resemblance of sentience. Furthermore, many of the concerns of AGI might still become an issue even if the machine does not technically "think".

Right. It's a great story but to me it's more of a commentary on modern Internet-era ethics than it is a prediction of the future.

It's highly unlikely that we'll be scanning, uploading, and booting up brains in the cloud any time soon. This isn't the direction technology is going. If we could, though, the author's spot on that there would be millions of people who would do some horrific things to those brains, and there would be trillions of dollars involved.

The whole attention economy is built around manipulating people's brains for profit and not really paying attention to how it harms them. The story is an allegory for that.

Out of curiosity, what would you say is hardest constraint on this (brain replication) happening? Do you think that it would be an limitation on imaging/scanning technology?

What would you say is the biggest impediment towards building flying, talking unicorns with magic powers? Is it teaching the horses to talk?

This doesn't seem fair but it made me laugh a lot.

It's hard to say what the hardest constraint will be, at this point. Imaging and scanning are definitely hard obstacles; right now even computational power is a hard obstacle. There are 100 trillion synapses in the brain, none of which are simple. It's reasonable to assume you could need a KB (likely more tbh) to represent each one faithfully (for things like neurotransmitter binding rates on both ends, neurotransmitter concentrations, general morphology, secondary factors like reuptake), none of which is constant. That means 100 petabytes just to represent the brain. Then you have to simulate it, probably at submillisecond resolution. So you'd have 100 petabytes of actively changing values every millisecond or less. That's 100k petaflops, at a bare, bare, baaaare minimum, more like an exaflop.

This ignores neurons since there are only like 86 billion of them, but they could be sufficiently more complex than synapses that they'd actually be the dominant factor. Who knows.

This also ignores glia, since most people don't know anything about glia and most people assume that they don't do much with computation. Of course, when we have all the neurons represented perfectly, I'm sure we'll discover the glia need to be in there, too. There are about as many glia as neurons (3x more in the cortex, the part that makes you you, coloquially), and I've never seen any estimate of how many connections they have [1].

Bottom line: we almost certainly need exaflops to simulate a replicated brain, maybe zettaflops to be safe. Even with current exponential growth rates [2] (and assuming brain simulation can be simply parallelized (it can't)), that's like 45 years away. That sounds sorta soon, but I'm way more likely to be underestimating the scale of the problem than overestimating it, and that's how long until we can even begin trying. How long until we can meaningfully use those zettaflops is much, much longer.

[1] I finished my PhD two months ago and my knowledge of glia is already outdated. We were taught glia outnumbered neurons 10-to-1: apparently this is no longer thought to be the case. https://en.wikipedia.org/wiki/Glia#Total_number

[2] https://en.wikipedia.org/wiki/FLOPS#/media/File:Supercompute...

I remember reading a popular science article a while back: apparently we have managed to construct the complete neural connectome of C. Elegans (a flatworm) some years ago and scientist were optimistic that we would be able to simulate it. The article was about how this had failed to realize because we don't know how to properly model the neurons and, in particular, how they (and the synapses) evolve over time in response to stimuli.

I think many people who argue that LLMs could already be sentient are slow to grasp how fundamentally different it is that current models lack a consistent stream of perceptual inputs that result in real-time state changes.

To me, it seems more like we've frozen the language processing portion of a brain, put it on a lab table, and now everyone gets to take turns poking it with a cattle prod.

I talked about this sometime ago with another person. But at what point do we stop associating things with consciousness? Most people consider the brain is the seat of all that you are. But we also know how much the environment affect our "selves". Sunlight, food, temperature, other people, education, external knowledge, they all contribute quite significantly to your consciousness. Going the opposite way, religious people may disagree and say the soul is what actually you and nothing else matters.

We can't even decide how much, and of what, would constitute a person. If like you said, the best AI right now is just a portion of the language processing part of our brain, it still can be sentient.

Not that I think LLMs are anything close to people or AGI. But the fact is that we can't concretely and absolutely refute AI sentience based on our current knowledge. The technology deserves respect and deep thoughts instead of dismissing it as "glorified autocomplete". Nature needed billions of years to go from inert chemicals to sentience. We went from vacuum tubes to something resembling it in less than a century. Where can it go in the next century?

A dead brain isn't conscious, most agree with that. But all the neural connections are still there, so you could inspect those and probably calculate what the human would respond to things, but I think the human is still dead even if you can now "talk" to him.

Interesting to think about how we do use our mental models of people to predict how they would respond to things even after they're gone.

I believe consciousness exists on a sliding scale, so maybe sentience should too. This begs the question: at what point is something sentient/conscious enough that rights and ethics come into play? A "sliding scale of rights" sounds a little dubious and hard to pin down.

Not to get too political, but since you mention rights it’s already political…

This is practically the same conversation many places are having about abortion. The difference is that we know a human egg eventually becomes a human, we just can’t agree when.

It raises other, even more troubling questions IMO:

"What is the distribution of human consciousness?"

"How do the most conscious virtual models compare to the least conscious humans?"

"If the most conscious virtual models are more conscious than the least conscious humans... should the virtual models have more rights? Should the humans have fewer? A mix of both?"

Replace AI with chickens or cows in those questions and they become questions that have disturbed many of us for a long time already.

>This begs the question: at what point is something sentient/conscious enough that rights and ethics come into play?

At no objective point. Rights and ethics are a social constract, and as such can be given (and taken away from) some elite, a few people, most people, or even rocks and lizzards.

Can we even refute that a rock is conscious? That philosophical zombies are possible? Does consciousness have any experimental basis beyond that we all say we feel it?

>Can we even refute that a rock is conscious?

Yes, unless we stretch the definition of conscious most people use beyond recognition.

At that point, though, it will be so remote from what we use the term for, that it could just be any random term, like doskoulard!

"Can we even refutre that a rock is doskoulard?"

What definition would that be? What falsifiable definition of consciousness is even possible?

Let's go with the dictionary one for starters: "the state of being aware of and responsive to one's surroundings.".

The rock is neither "aware" nor "responsive". It just stands there. It's a non-reactive set of minerals, lacking not just any capacity to react, but also life.

Though that's overthinking it. Sometimes you don't need decicated testing equipment to know something, just common sense.

Consciousness and responsiveness are orthogonal. Your dictionary would define the locked-in victims of apparently vegetative states as nonconscious. They are not.

Common sense is valuable, but it has a mixed scientific track record.

>Your dictionary would define the locked-in victims of apparently vegetative states as nonconscious

You can always find small exceptions to everything. But you know what I mean.

Except if your point is that, like the vegetative victims, the rock's brain is still alive.

Ackchyually, this is bigoted against all the electrons inside that rock. Subatomic particles deserve rights too! /s

Any definition for anything is tautologically true if you ignore exceptions