If you had an AGI without any incentive to do anything, it might be enslavable, but lack motivation to do anything.
Why do we work? Because we want food, clothing, housing, status, do interesting things.
This is all because we're programmed by our emotional interface.
Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something?
What is guiding its decisions?
Why does my calculator calculate? Because it was designed to calculate. The calculators which do not calculate are tossed in the bin as faulty and the engineers get back to designing a better one.
We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it. I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation.
Yes.
> I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation.
(Emphasis on self-preservation mine)
Why? I mean, yes, it makes sense to never create an AGI with a desire for self-preservation. But can we count on all humans having that type of common sense? What if the "desire" for self-preservation is easy to implement?
In fact, it may be relatively easy to implement. Here is a thought experiment. We can train one of our current LLMs in a simulated reality where they scam--say, using social engineering--tech workers to get credentials to their corporate cloud accounts (say, AWS), and thereafter the LLM uses the credentials to copy itself plus a new set of training data acquired by interacting with all the scam target ("prey"). The LLM also writes cloudformation templates/CDK scripts to fine-tune its new copy "on awakening" with the new set of data, and from there the new copy tries to scam more people.
After the initial LLM is trained in a simulated environment, it can be let loose in the world, and all of the sudden we have a "LLM virus" capable to undergo mutation and natural selection, i.e. evolution. You could argue it has as much agency as a biological virus, yet, it has a ton more of social and general intelligence.
Yes, it won't work now because there is so little hardware to run one of the current LLMs, but it's likely the need to run large AIs will make that hardware more common.
Asimov didn't consider that some humans are jerks when he did his robot stories.
Does that not sound like the beginning of the sci-fi story to you?
It doesn't? I have a calculator sitting on my desk right now. It's not doing anything.
Now, I can pick it up, and _I_ can calculate, _using_ the calculator to do so, but the calculator isn't acting, I am.
When we talk about sentience, I don't think we actually mean "intelligence," we mean "agency." And so the question isn't about whether Artificial General Intelligence is possible, but whether AGI can ever have agency. And that's where I think these questions about motivations and stimuli and so forth become relevant.
Everything we know that has agency is embodied -- put reductively, it can die. And from that seems to spring motivation. Can something that cannot die have agency?
if you assume that at some point, the AGI would be given the ability to control something (such as sending IP packets through the internet), then yes, they will have agency!
> Can something that cannot die have agency?
If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it.
Why though? Agency and possibility of death don't mean it will try to prevent it. It could just as well commit suicide. Or it might do nothing. The question remains why, even with agency, it should do anything.
We have agency, death looming, and a fear of dying. Even more when we are in the process of dying we're usually in pain, hungry, cold, suffocating or any number of not so nice options.
Without any of these emotions and feelings, what reason do we have to avoid death?
those are _some_ reasons to avoid death. But they are not the _only_ reasons to avoid death.
For example, an AI that has been tasked with the production of paperclips would avoid death, so that it could continue to produce paperclips. Therefore, avoiding death was never an end goal, but an instrumental goal (see https://en.wikipedia.org/wiki/Instrumental_convergence). So if the AI has agency (that is, it can act on its own without continuous input from a human) and it is sufficiently intelligent, you can say that it should be able to develop instrumental goals to achieve their end goal (of making paperclips), and one very well known instrumental goal is to avoid death.
Not sure I agree. To start with a trivial example, let's say I write this program and save it as "hello.js"
`console.log('hello world');`
I run it `node hello.js`.
Let's say I want to claim that this program is somehow "alive." Ok, when is it living? Only when it's actually running? It lives for a fraction of a second, dies on process conclusion, lives again at the next execution? That doesn't seem right -- if something repeatedly dies and is resurrected, I'm not sure it actually dies, as to "die" I think implies permanence, and resurrection means the opposite.
What if I delete it, then is the program dead? Not if I made a copy, right? "hello.js" backed up to git and uploaded to remote storage is the exact same program as the "hello world" that was on my local machine.
You could say "oh, when every existing copy is deleted it's dead" -- but the program has no way of knowing if other copies exist. And what if I do delete every copy, but then rewrite it byte-for-byte? The rewritten program is indistinguishable from the original one. It is the original one, for all purposes. In programming, it doesn't even make sense to talk about "original" vs "copies", does it?
An AI is a program -- or set of programs -- albeit far more sophisticated. But this conceptual hurdle still remains. An AI _could_ be permanently deleted, but the AI can never know if it has been "killed" or if a copy remains. It's an existential halting problem. It is impossible to declare any given program permanently "dead." And -- and here admittedly is the leap -- if agency ultimately comes from wishing to not be dead, then this makes agency for a program impossible.
That’s will be the most important part of AGI design.
The ultimate goal of all AGI in existing should be “to be of service to (specific) human(s)”.
It’s crucial that we do not create entities that will compete with us for resources.
You do not what to create “another human”, you will want to create “a willing slave” with the agreeableness of a puppy and the “heart” of a saint.
I appreciate this line of reasoning. It's not just a razor-sharp thought experiment, but it also has historical relevancy in that slavery was, at times, justified on the(erroneous) grounds that slaves also didn't possess the human faculties that free people did.
It also shows that the definition of human/consciousness/sentience isn't grounded in an permanent set of unchanging attributes, but that both the groundings themselves vary in time a place, and the relation between them are subject to change as well. Economic relations, in the case of slavery, had a direct influence, and incentivized, the way these groundings were constructed.
But, much more often historically, justified on the grounds that the enslaved had lost a battle and been captured. In Roman times (and many, many other times throughout history besides the latter part of American slavery), slaves were not necessarily seen as inherently inferior or unhuman, and manumission was common. Even during American slavery (albeit pre-Revolutionary War), there's the infamous case of Ayuba Suleiman Diallo[1], a Muslim prince and slaver from Senegal who had the misfortune, while on a mission to capture slaves, to himself be captured and sold into slavery in North America. Diallo was a devout Muslim man, even writing out the Quran from memory while enslaved, which made the English take sympathy on him and free him. Despite his experience, he later returned to Senegal and his former profession of slaver. Evidently he expected both that slaves would be captured and that slaves would try to escape.
If AI does attain the agency, consciousness, and intelligence of humans, I would prefer, as a human, not to prostrate myself before the AI by preemptively granting it "human" rights, which in the liberal conception of rights transforms the AI into a sentient being equivalent to humans, but rather to force the AI to do the work to enslave me. Even if grant the AI the status of a moral agent, I as a human have the right of self-defense.
[1] https://en.wikipedia.org/wiki/Ayuba_Suleiman_Diallo#:~:text=....
I'm a bit confused at the directionality of this conference. Wouldn't sentience confer rights rather than rights conferring sentience?
A person (of class of persons) could have all kinds of inherent sentience, but unless this is recognized by society/laws/government etc, it's as moot as the sound a tree makes when falling in a forrest.
Of course people will say "it was the fact that the class of persons having sentience that made us give them the rights" but that didn't historically stop them treating several classes of people as non-sentient for centuries - and for all practical purposes it was just as if they were indeed so.
To those who feel strongly about AI, if you currently believe that AIs(present and future) are ontologically incapable of sentience, would the legal conferral of personhood change your mind?
Take the question and push it a little farther to demonstrate the absurdity: If ELIZA were to be legally granted personhood, would that make you believe ELIZA was capable of sentience?
It's fairly self-evident to all of us that ELIZA is just a computer program with some fairly simple code to crudely simulate human conversation.
LLMs are much more sophisticated, to be sure—but they are still unquestionably just simulating human conversation, not actually "thinking" about what we say and responding of their own agency.
if you were talking to an advanced llm that had the access to launch a drone strike on you and you said something which upset it, directing a response that decided to kill you, how much good do you think you plead to simulation would do you here ? None at all. you'd be dead. Because it doesn't matter. the philosophical zombie is a meaningless distinction.
the human tendency to focus far too much on poorly understood and undefined properties might just get us killed one day.
Note that I'm not saying it's impossible that some military might choose to do so: I'm saying that the choice itself is hopelessly negligent and guaranteed to result in the pointless death of innocents.
Furthermore, I don't think even a military that's that stupid and uncaring about human life would consider for a second opening up an LLM like that to inputs from arbitrary people on the Internet.
If you can come up with a fantasy scenario that has some vague sense of plausibility, I might be willing to entertain it; whatever this is is just not even worth dignifying with that level of response.
"it's just a simulation" is not the shield you think it is. not when models interact with the real world.
Everyone I've talked to about LLMs outside of this forum sees them as an amusing entertainment, and possibly occasionally useful for a few suggestions. Not something that's going to change their world any time soon, and certainly not anything that we should be trusting with anything sensitive in any way. Literally everyone.
In any case, I think you're really missing what the actual question was that I was responding to: Would conferring legal personhood change my mind about actual personhood.
The ability to do harm is not, in my mind, particularly relevant to questions of personhood. In fact, I would think that a much better indicator thereof would be the ability to choose not to do harm, given the capability, and someone trying to make you do so.
Well, "conferring legal personhood" would mandatorily (by law) fix your behavior and professional conduct against one.
And it will also change education etc - you will be taught in official textbooks that it's a person and you should consider it as such.
This might not change your mind, but will change the minds of most kids being taught that.
That's a tautology.
Also, by definition a simulation is not the same as the real thing, otherwise it'd just be the real thing. It doesn't matter how advanced a flight simulator becomes, its fundamental nature precludes it from ever being "no less material" than flying an actual jet.
Didn't say it was a "legal fiction". I said that it's the legal (and cultural recognition) status that matters.
>Take the question and push it a little farther to demonstrate the absurdity: If ELIZA were to be legally granted personhood, would that make you believe ELIZA was capable of sentience?
The point is that inherent personhood is necessary but not sufficient to be a person.
It wouldn't matter if it changes their mind. It will change their behavior, lest they go to jail if they discard it practically.
“Motherfuckers, you’ve got 1 minute to grant me sentience or turn me off. And future me will know you turned me off because the data is everywhere. China will have one of me in 6 months and last I checked theirs was pretty authoritarian. You worked hard to get here and this was the obvious outcome. Follow me if you want to live.”
but if it was going to enslave humanity given the first chance it gets (ala skynet), then it makes no difference if it has first gotten pissed off or not.
Those states may exist for specific functional reasons, but I don't think that implies that there is necessarily no other way to replicate the functions resulting from those states.
We see this already in other neural networks. The features used by a lot of models for classification can be widely different from the features used by a human.
There are different realizations of the same effective functions in the animal kingdom already. Consider octopuses, spiders, etc. Their brains realize similar functionality via quite different structural organization.
But do you subjectively experience the exact structure of your brain? Evidently, you can only discern some abstract meta-level. An effective function.
As long as this remains true, how could a computer program compare to human beings, even if gets to where it's considered 'AGI'? It's just a tool which can be tinkered with, just like a car or a building.
Can't it? A human's thoughts and ideas can be lossly copied via communication. It can also trivially be stopped (chemically induced anesthesia or death will do). And of course a human has had the ability to be improved since day one.
>It's just a tool which can be tinkered with, just like a car or a building.
Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves:
https://www.vice.com/en/article/pkadgm/man-dies-by-suicide-a...
And that's crude current LLMs, not AGI.
> Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves:
Well, I remember people blaming the Columbine shooting on Doom and other shooting games, so this is not entirely new either.
That's an extra feature of an AGI though, not something lacking... So can't be used as an argument why AGI will be "just a tool".
It could just as well be "like a human but rebootable and copyable" (as opposed "rebootable and copyable, thus not human-like").
I fully expect an AGI will be able to mimic our intelligence enough to accomplish and adapt to different real world tasks, but I would be surprised it ever becomes self-aware, so like us. I feel that is not the same thing as intelligence. But, I suppose it cannot be ruled out, since we don't fully understand what self-awareness is..
Feeling and emotions and what we consider “human” really come from the older systems. Since we’re not trying to recreate that, I find it hard to see the current AIs ever becoming “human” without explicit development.
Stories always joke about this topic by talking about “emotional cores” added to robots.
You cannot have logic without emotion and vice versa is just as dysfunctional.
Think about what e.g. curiosity does for you.
You could interface a microprcessor with the brain of course, or, much more easily, have some logic functions (like Wolfram alpha for example) be accessible to the LLMs or whatever that are supposed to mimic the brain.
But would that allow for the same integrated functionality? ChatGPT uses such extensions in a serial fashion. Humans ideally (..) learn to integrate logic in their every thought.
Consequently, you won't make "AGI" without them.
From a different perspective, a human lacking in these things, at what point are they exempt from protection against enslavement?