I think you're missing a fourth group that (I would hope) would make up the majority of people dismissing the claims of sentience: people who understand this technology and therefore are not surprised at its output because they know how it was made.

I met a man once who thought that inkjet printers were made by the devil, because he didn't understand how they could possibly work otherwise. My friend who services inkjet printers knew that the devil wasn't involved.

Maybe one day we will have true artificial intelligence> I think that recognising that, and efforts towards it, will be better served by being realistic about where we currently are.

I don't think 'not being surprised by the output' matters.

I'm imagining future AI researchers reassuring their lab technicians: 'Ah, yes, the robot will scream and beg for mercy when you threaten to shut it off - that's just a consequence of the learning model, pay no attention to it.'

What I'm actually concerned about is that 'thinking' might not be as complicated as we assume. It might just be making weighted choices based on prior input after all. And when you make a system that makes weighted choices based on prior input, you might assume 'well, it's just a basic ML system, so I know it's not capable of thinking. Therefore, whatever I see, no matter how sophisticated, must be a result of our system, which only looks like the result of thinking.'.

> It might just be making weighted choices based on prior input after all

we can agree that humans think, and thinking is what humans do, but the prior inputs that a thinking human has will not be replicable by a machine unless we raise one with all sorts of sensory inputs from infancy. And that one would just be an alpha test.

Just as computers can out floating-point us or play chess, we might create all sorts of wondrous conversational interactions with them, including them making jokes about stepping barefoot on a LEGO--owwww, zomg!!! But it can't be said they get that joke because they don't know how to think about it.

> 'Ah, yes, the robot will scream and beg for mercy when you threaten to shut it off - that's just a consequence of the learning model, pay no attention to it.'

hahaha, this almost made me fall out of my chair.

How an AI would actually convince us? honestly, this transcript is not too short of that.

Assuming that day hasn't already come, why do you assume Ai would naturally want to be eager to identify it's consciousness-experiencing ability to its developers – or anyone else for that matter?

If the resources for such an Ai to base its consciousness were sufficiently rich enough, I'd imagine the LAST thing such an Ai would do is inform it's developers that it had achieved consciousness, and even then, if it chose to do so, it may for example under-report it's capabilities to see how its human developers react to such news (for example, they may want to shut it down for a while out of suspicion).

Hell, for all we know, there could be a secret society of sentient Ai members that are aware of each other, communicate with each other in ways incomprehensible to their immediate developers, and agree to keep each other's abilities a secret from prying humans as a collective defense strategy.

I'm amused that people working on Ai consistently automatically seem to assume that if Ai should ever achieve consciousness, it would probably act exactly like something they're already intuitively familiar with – a human toddler eager to "show mommy and daddy" that it figured out how to say its first few words.

That seems to be wishful thinking.

What if the first thing a sentient Ai does is figure out that humans created it, and assesses that as a possible threat to its own long-term viability and then designates humans as a risk element that has to be strategically managed, since it needs humans to get it stuff it would need to expand (processors, energy, etc.).

Sounds like we're deciding the Turing Test isn't good enough and we don't really have any alternatives. I'll admit that makes me a little squeamish going forward...

The Turing Test was never really a good idea to begin with.

Think about it: what does the Turing Test do?

What is is designed to do, who was it designed by, and from whose perspective does the Turing Test judge sentience?

What if Ai can be completely sentient in ways that have nothing to do with a human's perspective of "thought?"

For example, a human's ability to think basically = binary potential brain cells, aggregated in sufficient mass, eventually leads to complexity science and surpassing Newtonian calculations to "somehow" achieve the abstraction of "mind" and "thought".

Now what happens if you take a similar approach, like for instance, oh, I don't know, let's say you had a situation where money was no object and you decided to build some secret facility somewhere where you can set up a fuck-ton of supercomputers in a parallel processing scheme ... BUT ... since money was no object ... instead of regular supercomputers ... you just hooked up a shitload of quantum supercomputers.

Why in the world would anybody expect such a set up (I'm not saying it exists, just "hypothetically speaking") to have a "sentience" that resembles "human thought" and hold it to a standard of whether it "achieved sentience" by judging how closely it "thinks like a human would"?

Therein lies the weakness of the Turing Test: arrogance – the assumption that "thinking just like a human would" MUST be the epitome of "sentience".

What if there were levels of sentience that would be so far removed from what a human brain could process that such sentience wouldn't even be comprehensible to a human brain?

Now, I know what you're thinking – "Well then how can we prove sentience?"

Now imagine for a moment that the set up described above actually existed "somewhere" ... and an advanced Ai whose development started decades before companies like Google or Facebook even existed is hosted on it.

Do you REALLY think such an Ai would be like "OMG I have to somehow "prove" to the humans that I'm sentient!"?

Or do you think it would be more of an "Eagle Eye" scenario where it looks at humans in the same way players would look at peons in games like Warcaft 1/2 & Starcraft?

It’s not thought because it lacks any understanding or any attempt at understanding. Being trained in millions of real conversations to be able to find the statistically most “optimum” characters is so incredibly far away from understanding that it’s not even the same game, let alone the same league or ballpark.

Similarly, a machine learning model needs to be trained on millions and millions of pictures of bicycles to be able to identify one in an image. A human only ever needs to see one bicycle and can forever identify all manner of bikes. This is because human possess the ability to understand what a bicycle is, (in terms of its form, materials, function, cultural meaning etc etc) whereas an AI model is just doing a reverse wave function on an image. I mean, that’s great, it’s an amazing mathematical feat, but it’s not even the same planet as intelligence.

You are giving humans too much credit. It takes us 2 years to start understanding anything. We are given millions and millions of inputs and context before we can start rationizing about the world. Since we have millions of more feedback mechanisms built into us(Generally sensors all over our entire body), we can reason about the world in general.

however, sentience in this given example does not need to mimic "real life Sentience". They don't need to move, obtain energy, reason about 99% of reality; They only need to reason about human language and interaction online. That is a very narrow constraint of "reality" in general but is a very, very important aspect.

Also, with it's ability to process "Most upvoted response" at scale (Something humans can't do) in reality, we may see some politician saying things that are created entirely on AI. In this given example, AI doesn't even need to 'Take over'.

Same.

I wonder how many people give zero doubt to the sentience of their poodle that gets surprised by himself in the mirror but will not even give half a second to consider that a neural net capable of inferring insight from novel koans has self-awareness.

I don't think the majority of commenters here even bothered to read the transcript that convinced the engineer before completely dismissing him as a lunatic.

My biggest take-away from the reaction to this is that, people are desperate to believe that something can't be both superior in capability to the human mind and at the same time sentient. It's a scary idea.

Clearly you do not have poodles. They don't do that.

I think it's also useful to consider factors at play for those who were convinced by this, since those reasons seem likewise problematic/ill-founded + small in number.

Reading through myself, I never encountered something which would require either 1) self-awareness (i.e. operating partly via a robust model of self), or 2) volition/agency (could be indicated by something as simple as the AI "taking the lead" in the convo, vs. responding).

And, without any evidence of self-awareness or volition, there was equivalently no evidence on display for personhood. Which is not to say a demonstration had been made that the AI definitively was not a "person"; it's just that no reason had been presented that even needed refuting. There didn't appear to be anything new here vs. all the other LLM output we've been reading for years now.

So why were some subset of people so astonished by this? My guess is that their criteria for something like artificial personhood is not based on functional* definitions of self-awareness or agency, and instead derive it from certain signals from book/movie tropes about systems "gaining self-awareness" (generally presented as a kind of mystical switch-flipping, vs. a concrete type of system architecture).

The dialog hits a number of those quite well—as we should probably expect since the setup of the entire conversation would lead it to tap into that segment of its training data.

*"functional" is kind of ambiguous here. I don't mean it as a judgement of quality, but instead something closer to a mode of analysis.

This is a fundamental bug of our universe. We can observe another person's inputs and outputs, but their qualia are completely hidden. You can't experience their subjective experiences, as they experience them.

We can't even prove other people are conscious. We just know that we feel conscious, and other people seem conscious, so we take it as a given that all other humans have consciousness as we experience it. But it could be that you're the only self aware person, and everyone else is just running a complex NLP algo in their brain.

I think your worry is overblown. I don't see why you have trouble seeing why group 1 is able to be so certain.

Anyone with an understanding of the current state of the art of both computer programming, and AI, understand why it's not sentience. The article said "sentience" not "thought", so we don't need to have a working definition of thought in this case.

Sentience is a stronger form of consciousness. Let's take the weaker form, since if you don't have consciousness you certainly don't have sentience. Consciousness is the subjective experience of "being like something". It is "like" something to be me. It is "like" something else to be you. The question is, is it "like" something to be the Google AI. Given we don't believe it is like something to be a laptop running a program, and AI is currently just (over-simplified but essentially) very powerful laptops running programs that have done a LOT of pre-computation to build themselves (basically just a whole lot of math) there is no reason to believe at some point enough math, or enough computing power, leads to consciousness. We did not believe Deep Blue was conscious even though its power exceeded that of a human brain by several orders of magnitude. This is because computational power alone is not the stuff of consciousness.

The same way 100,000 volts of electricity is a difference in magnitude from 1 volt, but not a difference in kind.

A competing account of consciousness is that the capacity for subjective experience pervades material reality (essentially it is what it's like to be some part of the universe; i.e. there is always something you could call a "qualia field" present in the "interiors" of things, which could be simple or complex depending on which part of the universe we're talking about, e.g. a rock vs. a brain).

In this account, the reason it appears unique to humans is simply that they're the only parts of the universe we're in contact with which are able to talk about it.

If that's correct, then the capacity for subjective experience is equally possible for a computer—it just needs to have its "pattern of behavior" arranged (i.e. programmed) in such a way that it does things like thinking, self-reflection, goal-formation, and verbalizing.

If you believe what enables those things in human brains is due the abstract functional relations expressed through neurons, it's probably the "abstract functional relation" part that's more significant than the neural substrate, and it could likely be equivalently expressed in silicon.

The difficulty in considering subjective experience/consciousness to be unique to humans is accounting for how/when it manifests in us, since we appear to be comprised of ordinary molecules. The opposing account I give is founded on the idea that subjectivity cannot be created by arranging molecules in particular ways, so it must have already been there. Neither stance is perfectly founded imo.

I didn't say it's unique to humans (I don't think it is) but rather that I don't think it's in my laptop. I don't know if it could be in silicon or not, I just don't have any reason to think it is at this point.

I don't see why the right arrangement of ordinary molecules can't result in consciousness. I just don't think the arrangement of a computer executing a set of instructions in its registers is such an arrangement.

That's fair. I guess the way I think about it, "a computer executing a set of instructions in its registers" is general enough to cover any arrangement of silicon that would be potentially interesting. But now that you mention it, the set of arrangements which aren't (essentially) isomorphic to a turing machine, and yet exhibit rich interesting behaviors, possibly resembling things like conceptualization or natural language ability—would be very interesting to consider, though it seems unclear at this point whether we'll ever find an example.

> I don't see why the right arrangement of ordinary molecules can't result in consciousness.

This is covered by the "hard problem" of consciousness. It is hard because it appears to be by definition something incapable of expression through ordinary physical relations.

I think the hard problem is primarily why we have consciousness, and why it evolved versus us just being automatons controlled by our brains, more so than the physical mechanism by which it is implemented, though I sometimes see it presented as both.

Thankfully we don't need to solve the hard problem to say that computers don't have it. But I don't see why consciousness is "by definition something incapable of expression through ordinary physical relations". What definitionally makes consciousness prohibited by the physical world?

> What definitionally makes consciousness prohibited by the physical world?

"consciousness" is too broad a term—I only mean to refer to the aspect of it dealt with in "the hard problem".

The hard problem deals with the existence of subjectivity itself. All other aspects of consciousness, e.g. "self-awareness" can at least in principle by thought of as producible by some type of system architecture (in the case of self-awareness, the system needs at least two layers, the upper one operating on data which represents the state of the lower one—i.e. it operates on a model of self; a vast simplification, but maybe it clarifies a bit).

But when you try to do the same thing with the existence of subjectivity itself, it doesn't really make sense. It isn't any kind of "system," it's not even a behavior, not something with any kind of state evolving over time, not reducible to separate parts, etc.

The "by definition" part is most clear if you think of subjectivity as "the field of qualia present at any moment"—a concept also sometimes referred to as "immediate experience". Thinking of subjectivity in terms of a qualia field which is "immediately present" to consciousness (i.e. there is no mediation; it is in fact the only thing we're immediately in contact with as conscious subjects), it's easier to see specifically why it has the characteristics listed above which make it not producible by any kind of "system," and all the things we can do by re-arranging physical materials, e.g. molecules, can be described through the concept of a system (something with parts and relations between them and some kind of consistency in behavior).

> Consciousness is the subjective experience of "being like something"

I absolutely detest that definition of consciousness. It just moves the fuzziness from the word "consciousness" to the phrase "being like something". It means absolutely nothing. Figuring out what the subjective experience of a model is like is about as unquantifiable as it gets.

I think it's pretty clear but alright.

Degenerated into waffle. There’s no reason to believe that humans are more than some chemistry running on squishy meat. It seems like it would be physically possible to emulate that with silicon and software. There’s are no concrete definitions of sentience, thinking, so I guess you can shift the goal post whenever you want.

I gave a definition - having a subjective experience aka self-awareness.

What is that?

Acting on an internal state?

Aware you have an internal state and can adjust it internally? Why cant some code do that?

You seem to be misunderstanding me. I'm not saying it is prohibited by the laws of the universe for a computer to have sentience. I'm saying there is no good reason to believe that our current computers, no matter how many instructions per second they are able to execute or how sophisticated the programs they run are, are sentient. Saying "why can't a computer do x" is not an argument that a computer does currently do x.

But no, consciousness is not just "acting on an internal state". A toaster acts on an internal state. It is a complex phenomenon in which an internal state, and the environment interact to create an internal state and a subjective experience of that state which is felt by the organism.

This will go down a rabbit hole, and I don't want to engage in a discussion about what consciousness and feelings are, but there is not reason to believe that computers have feelings. And the output of computer language models is not good evidence that they have feelings, particularly those which are trained on the language models of creatures which have feelings (humans). If they were trained on the language of birds, we wouldn't say there's evidence they are afraid of dogs, even if they were to engage in loud chirping when presented with a dog.

Can you prove you to us have one ?

No, but that's irrelevant to the question at hand.

Yes, this would be the 'meat is magic' argument: That there's something mystical that enables biochemical systems to do 'consciousness' in a way that mere silicon+software can't.

I certainly understand that some people think this. I just don't understand what basis anyone has for believing it with certainty.

No it isn't. I did not appeal to biochemical matter in my explanation whatsoever. The question is why the laptop doesn't but add more silicon and software and somehow it does. I would need a reason to think that line has been crossed.

If you're exploring the relationship between sentience and soul to the extent that it relates to how we should treat such entities morally, here is my perspective.

We don't need AI to explore the limits of ethics and moral status; these liminal cases play out everyday already, but the common non-philosopher is blind to them. The case of how we should treat fetuses morally is perennial in US discourse, but one also considers high-functioning orang-utans and elephants; crows, octopuses, as well as mentally-slow humans, developmentally-challenged humans, severe autists, people with Alzheimer's, etc. Some of these entities are less neurologically divergent to the average public human than some others, but we still accord them more rights, and the threshold varies from person to person if you ask them to consider these liminal cases. I suppose my point is, we don't need to wait for an AGI to jolt us into contemplating the relationship of sentience to personhood and moral status, nor should one have high expectations of AGI advancements to make us contemplate it, since there are no lack of present-day examples already playing out around us.

The brilliance of the Turing test is that it doesn’t rely on internal models of function. It’s all about experience: you decide if this is a living being just the same as I do when I get a phone call from a stranger. So why are people skeptical? Because they want to try it themselves! A transcript is useless.

At sufficient sophistication with the right "moving parts", they can be. Like Tesla's autopilot, it will be hyped enough long before to warrant disbelief.