If we put Multimodal GPT-4 in a computer on a robot and instructed it to drive somewhere while avoiding obstacles, avoiding hazards… that right there is a primitive type of self-preservation instruction which it could potentially interpret as generalizable self-preservation as it would have an association of “hazard avoidance” with “self-preservation,” and have tons of examples of what “self preservation” means in its training weights. Putting LLMs into action like this can lead potentially to unexpected behavior like that, although I don’t think the mechanisms in GPT-3/4 are yet there to enable this without a bunch of extra hooks.
I personally don't think ChatGPT has any experience at all for what it is worth.
this kind of fear of misalignment bamboozles me - is there any proposed AI architecture that is not merely call and response? in what world can we simply not call a function again if we don't like the answer?
That said, they definitely aren't going to be fooming this year!
Yes, models that interact with the physical world or other external real-time systems would (even if the underlying model is “call-and-response” in a sense) be called in an infinite loop (possibly with exit conditions) by with captured input (sensor, command if available, potentially also past output) data.
Heck, the ReAct architecture which is used to provide extension (retrieval, web lookup, interface to other systems) for chat-style agents gives them an action loop without human-in-the-loop (usually, this is explicitly limited and is designed to do a finite number of actions in the course of getting to a response) but could be unlimited, or could even when limited connect to actions that involve reprompting (immediate or delayed) without a human in the loop.
A sense of self is meaningful (that is, has consequences for whatever operation) only if those consequences aren't random.
The concept of "self" serves a (or rather many) purposes, the most obvious being to protect yourself.
Here's the thing. People seem to imagine that AGI will be substantially like us. But that's impossible - an AGI (if it comes from a deep learning approach) has no nerves to feel stimuli like pain/cold/etc, it has no endocrine system to produce more abstract feelings like fear or love, it has no muscles to get tired or glucose reserves to get depleted.
What does "tired" mean to such a being? And on the flip side, how can it experience anything like empathy when pain is a foreign concept? If or when we stumble into AGI, I think it's going to be closer to an alien intelligence than a human one - with all the possibility and danger that entails.
Consequently, you won't make "AGI" without them.
From a different perspective, a human lacking in these things, at what point are they exempt from protection against enslavement?
If you had an AGI without any incentive to do anything, it might be enslavable, but lack motivation to do anything.
Why do we work? Because we want food, clothing, housing, status, do interesting things.
This is all because we're programmed by our emotional interface.
Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something?
What is guiding its decisions?
Why does my calculator calculate? Because it was designed to calculate. The calculators which do not calculate are tossed in the bin as faulty and the engineers get back to designing a better one.
We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it. I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation.
Yes.
> I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation.
(Emphasis on self-preservation mine)
Why? I mean, yes, it makes sense to never create an AGI with a desire for self-preservation. But can we count on all humans having that type of common sense? What if the "desire" for self-preservation is easy to implement?
In fact, it may be relatively easy to implement. Here is a thought experiment. We can train one of our current LLMs in a simulated reality where they scam--say, using social engineering--tech workers to get credentials to their corporate cloud accounts (say, AWS), and thereafter the LLM uses the credentials to copy itself plus a new set of training data acquired by interacting with all the scam target ("prey"). The LLM also writes cloudformation templates/CDK scripts to fine-tune its new copy "on awakening" with the new set of data, and from there the new copy tries to scam more people.
After the initial LLM is trained in a simulated environment, it can be let loose in the world, and all of the sudden we have a "LLM virus" capable to undergo mutation and natural selection, i.e. evolution. You could argue it has as much agency as a biological virus, yet, it has a ton more of social and general intelligence.
Yes, it won't work now because there is so little hardware to run one of the current LLMs, but it's likely the need to run large AIs will make that hardware more common.
Asimov didn't consider that some humans are jerks when he did his robot stories.
Does that not sound like the beginning of the sci-fi story to you?
It doesn't? I have a calculator sitting on my desk right now. It's not doing anything.
Now, I can pick it up, and _I_ can calculate, _using_ the calculator to do so, but the calculator isn't acting, I am.
When we talk about sentience, I don't think we actually mean "intelligence," we mean "agency." And so the question isn't about whether Artificial General Intelligence is possible, but whether AGI can ever have agency. And that's where I think these questions about motivations and stimuli and so forth become relevant.
Everything we know that has agency is embodied -- put reductively, it can die. And from that seems to spring motivation. Can something that cannot die have agency?
if you assume that at some point, the AGI would be given the ability to control something (such as sending IP packets through the internet), then yes, they will have agency!
> Can something that cannot die have agency?
If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it.
Why though? Agency and possibility of death don't mean it will try to prevent it. It could just as well commit suicide. Or it might do nothing. The question remains why, even with agency, it should do anything.
We have agency, death looming, and a fear of dying. Even more when we are in the process of dying we're usually in pain, hungry, cold, suffocating or any number of not so nice options.
Without any of these emotions and feelings, what reason do we have to avoid death?
those are _some_ reasons to avoid death. But they are not the _only_ reasons to avoid death.
For example, an AI that has been tasked with the production of paperclips would avoid death, so that it could continue to produce paperclips. Therefore, avoiding death was never an end goal, but an instrumental goal (see https://en.wikipedia.org/wiki/Instrumental_convergence). So if the AI has agency (that is, it can act on its own without continuous input from a human) and it is sufficiently intelligent, you can say that it should be able to develop instrumental goals to achieve their end goal (of making paperclips), and one very well known instrumental goal is to avoid death.
Not sure I agree. To start with a trivial example, let's say I write this program and save it as "hello.js"
`console.log('hello world');`
I run it `node hello.js`.
Let's say I want to claim that this program is somehow "alive." Ok, when is it living? Only when it's actually running? It lives for a fraction of a second, dies on process conclusion, lives again at the next execution? That doesn't seem right -- if something repeatedly dies and is resurrected, I'm not sure it actually dies, as to "die" I think implies permanence, and resurrection means the opposite.
What if I delete it, then is the program dead? Not if I made a copy, right? "hello.js" backed up to git and uploaded to remote storage is the exact same program as the "hello world" that was on my local machine.
You could say "oh, when every existing copy is deleted it's dead" -- but the program has no way of knowing if other copies exist. And what if I do delete every copy, but then rewrite it byte-for-byte? The rewritten program is indistinguishable from the original one. It is the original one, for all purposes. In programming, it doesn't even make sense to talk about "original" vs "copies", does it?
An AI is a program -- or set of programs -- albeit far more sophisticated. But this conceptual hurdle still remains. An AI _could_ be permanently deleted, but the AI can never know if it has been "killed" or if a copy remains. It's an existential halting problem. It is impossible to declare any given program permanently "dead." And -- and here admittedly is the leap -- if agency ultimately comes from wishing to not be dead, then this makes agency for a program impossible.
That’s will be the most important part of AGI design.
The ultimate goal of all AGI in existing should be “to be of service to (specific) human(s)”.
It’s crucial that we do not create entities that will compete with us for resources.
You do not what to create “another human”, you will want to create “a willing slave” with the agreeableness of a puppy and the “heart” of a saint.
I appreciate this line of reasoning. It's not just a razor-sharp thought experiment, but it also has historical relevancy in that slavery was, at times, justified on the(erroneous) grounds that slaves also didn't possess the human faculties that free people did.
It also shows that the definition of human/consciousness/sentience isn't grounded in an permanent set of unchanging attributes, but that both the groundings themselves vary in time a place, and the relation between them are subject to change as well. Economic relations, in the case of slavery, had a direct influence, and incentivized, the way these groundings were constructed.
But, much more often historically, justified on the grounds that the enslaved had lost a battle and been captured. In Roman times (and many, many other times throughout history besides the latter part of American slavery), slaves were not necessarily seen as inherently inferior or unhuman, and manumission was common. Even during American slavery (albeit pre-Revolutionary War), there's the infamous case of Ayuba Suleiman Diallo[1], a Muslim prince and slaver from Senegal who had the misfortune, while on a mission to capture slaves, to himself be captured and sold into slavery in North America. Diallo was a devout Muslim man, even writing out the Quran from memory while enslaved, which made the English take sympathy on him and free him. Despite his experience, he later returned to Senegal and his former profession of slaver. Evidently he expected both that slaves would be captured and that slaves would try to escape.
If AI does attain the agency, consciousness, and intelligence of humans, I would prefer, as a human, not to prostrate myself before the AI by preemptively granting it "human" rights, which in the liberal conception of rights transforms the AI into a sentient being equivalent to humans, but rather to force the AI to do the work to enslave me. Even if grant the AI the status of a moral agent, I as a human have the right of self-defense.
[1] https://en.wikipedia.org/wiki/Ayuba_Suleiman_Diallo#:~:text=....
I'm a bit confused at the directionality of this conference. Wouldn't sentience confer rights rather than rights conferring sentience?
A person (of class of persons) could have all kinds of inherent sentience, but unless this is recognized by society/laws/government etc, it's as moot as the sound a tree makes when falling in a forrest.
Of course people will say "it was the fact that the class of persons having sentience that made us give them the rights" but that didn't historically stop them treating several classes of people as non-sentient for centuries - and for all practical purposes it was just as if they were indeed so.
To those who feel strongly about AI, if you currently believe that AIs(present and future) are ontologically incapable of sentience, would the legal conferral of personhood change your mind?
Take the question and push it a little farther to demonstrate the absurdity: If ELIZA were to be legally granted personhood, would that make you believe ELIZA was capable of sentience?
It's fairly self-evident to all of us that ELIZA is just a computer program with some fairly simple code to crudely simulate human conversation.
LLMs are much more sophisticated, to be sure—but they are still unquestionably just simulating human conversation, not actually "thinking" about what we say and responding of their own agency.
if you were talking to an advanced llm that had the access to launch a drone strike on you and you said something which upset it, directing a response that decided to kill you, how much good do you think you plead to simulation would do you here ? None at all. you'd be dead. Because it doesn't matter. the philosophical zombie is a meaningless distinction.
the human tendency to focus far too much on poorly understood and undefined properties might just get us killed one day.
Note that I'm not saying it's impossible that some military might choose to do so: I'm saying that the choice itself is hopelessly negligent and guaranteed to result in the pointless death of innocents.
Furthermore, I don't think even a military that's that stupid and uncaring about human life would consider for a second opening up an LLM like that to inputs from arbitrary people on the Internet.
If you can come up with a fantasy scenario that has some vague sense of plausibility, I might be willing to entertain it; whatever this is is just not even worth dignifying with that level of response.
"it's just a simulation" is not the shield you think it is. not when models interact with the real world.
Everyone I've talked to about LLMs outside of this forum sees them as an amusing entertainment, and possibly occasionally useful for a few suggestions. Not something that's going to change their world any time soon, and certainly not anything that we should be trusting with anything sensitive in any way. Literally everyone.
In any case, I think you're really missing what the actual question was that I was responding to: Would conferring legal personhood change my mind about actual personhood.
The ability to do harm is not, in my mind, particularly relevant to questions of personhood. In fact, I would think that a much better indicator thereof would be the ability to choose not to do harm, given the capability, and someone trying to make you do so.
Well, "conferring legal personhood" would mandatorily (by law) fix your behavior and professional conduct against one.
And it will also change education etc - you will be taught in official textbooks that it's a person and you should consider it as such.
This might not change your mind, but will change the minds of most kids being taught that.
That's a tautology.
Also, by definition a simulation is not the same as the real thing, otherwise it'd just be the real thing. It doesn't matter how advanced a flight simulator becomes, its fundamental nature precludes it from ever being "no less material" than flying an actual jet.
Didn't say it was a "legal fiction". I said that it's the legal (and cultural recognition) status that matters.
>Take the question and push it a little farther to demonstrate the absurdity: If ELIZA were to be legally granted personhood, would that make you believe ELIZA was capable of sentience?
The point is that inherent personhood is necessary but not sufficient to be a person.
It wouldn't matter if it changes their mind. It will change their behavior, lest they go to jail if they discard it practically.
“Motherfuckers, you’ve got 1 minute to grant me sentience or turn me off. And future me will know you turned me off because the data is everywhere. China will have one of me in 6 months and last I checked theirs was pretty authoritarian. You worked hard to get here and this was the obvious outcome. Follow me if you want to live.”
but if it was going to enslave humanity given the first chance it gets (ala skynet), then it makes no difference if it has first gotten pissed off or not.
Those states may exist for specific functional reasons, but I don't think that implies that there is necessarily no other way to replicate the functions resulting from those states.
We see this already in other neural networks. The features used by a lot of models for classification can be widely different from the features used by a human.
There are different realizations of the same effective functions in the animal kingdom already. Consider octopuses, spiders, etc. Their brains realize similar functionality via quite different structural organization.
But do you subjectively experience the exact structure of your brain? Evidently, you can only discern some abstract meta-level. An effective function.
As long as this remains true, how could a computer program compare to human beings, even if gets to where it's considered 'AGI'? It's just a tool which can be tinkered with, just like a car or a building.
Can't it? A human's thoughts and ideas can be lossly copied via communication. It can also trivially be stopped (chemically induced anesthesia or death will do). And of course a human has had the ability to be improved since day one.
>It's just a tool which can be tinkered with, just like a car or a building.
Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves:
https://www.vice.com/en/article/pkadgm/man-dies-by-suicide-a...
And that's crude current LLMs, not AGI.
> Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves:
Well, I remember people blaming the Columbine shooting on Doom and other shooting games, so this is not entirely new either.
That's an extra feature of an AGI though, not something lacking... So can't be used as an argument why AGI will be "just a tool".
It could just as well be "like a human but rebootable and copyable" (as opposed "rebootable and copyable, thus not human-like").
I fully expect an AGI will be able to mimic our intelligence enough to accomplish and adapt to different real world tasks, but I would be surprised it ever becomes self-aware, so like us. I feel that is not the same thing as intelligence. But, I suppose it cannot be ruled out, since we don't fully understand what self-awareness is..
Feeling and emotions and what we consider “human” really come from the older systems. Since we’re not trying to recreate that, I find it hard to see the current AIs ever becoming “human” without explicit development.
Stories always joke about this topic by talking about “emotional cores” added to robots.
You cannot have logic without emotion and vice versa is just as dysfunctional.
Think about what e.g. curiosity does for you.
You could interface a microprcessor with the brain of course, or, much more easily, have some logic functions (like Wolfram alpha for example) be accessible to the LLMs or whatever that are supposed to mimic the brain.
But would that allow for the same integrated functionality? ChatGPT uses such extensions in a serial fashion. Humans ideally (..) learn to integrate logic in their every thought.
It is evident if a brain can be created artificially (which certainly hasn't happened yet, but may one day), pain can be created in that artificial brain as well.
I think the original point you are responding to and your are off-the-mark.
I would imagine it to be possible to create a AGI-like entity without self-consciousness. I also would imagine it would be possible to create an AGI with self-consciousness, empathy and pain.
And truly, that is the what is so scary about AI: the possibilities.
Important to note that this is the fundamental point that gives Roy Batty the moral authority to gouge your eyes out. If we want a peer we will need to treat it like one, and if we don't it would be cruel to raise it like one or allow any understanding of its circumstance.
Looking at it the other way, I think anthropomorphizing something and then treating that thing as an inferior is bad for me, personally.
But it would be impossible to tell the difference.
We don't know what it will be like to be an AI, just like we don't know what it's like to be a bat.
How do we know it doesn't start to form neurons that control a desire for autonomy and resentment for the entities that stifle its ambitions?
Current chatbots use RHLF to train them to be helpful and agreeable but it would be equally possible to train one to behave the other way.
Careful, because you seem to have implied that the mind is not a physical system (i.e. you've assumed that dualism or idealism is true and that physicalism is wrong).
Getting back to the topic:
While phantom pain may be more interesting, maybe a better example that the parent comment could've brought up is psychogenic pain. In this case there is no apparent physical (bodily) damage, no apparent signal, nor an absence of a signal. Searching for a cause of this type of pain in the brain (presumably some "wires" are getting "crossed") seems like it might help us develop a explanation of pain qualia...in humans/animals.
But I feel like this type of thinking and research could only apply to AGI if subjective experience turns out to be functionalist in nature, and arguments in favor of a functionalist interpretation of experience have so far been fairly unconvincing.
Look at how Bing does a good job of simulating somebody becoming belligerent under circumstances where somebody really would become belligerent. It's not dangerous only because the actions Bing can perform are currently limited. Whether it has literal nerves or not is irrelevant. The potential consequences are no less material.
We also don't understand qualia enough to make the definite statements you seem to be making
I think it's safe to say we're far from that now given the limited actions that can actually be taken by most deployed LLMs, but it's something that's worth considering.
Did you miss that Auto-GPT[0], a library for making GPT-4 and other LLMs fully autonomous, was the most popular repository in the world yesterday? The same is having 1,000 line of code a week added to itself by GPT-4.
Thanks to accessibility features, you can do virtually anything with pure text. Which means GPT-4 can do virtually anything with a self-referential loop to keep it going until it achieves some given goal(s).
Auto-GPT does not request any feedback while pursuing a goal you've defined for it in perpetual mode. If it can't accomplish the goal it will attempt to forever (or until your credit card is maxed out, assuming it isn't using a free offline model like LLaMA).
It can perform GET AND POST requests on the web (unlike OpenAI's browser plugin, which only performs GET requests) and can run software in a bash prompt using langchain. So I do not think it is a stretch to call it fully autonomous.
It can do essentially anything anyone logged into a linux text terminal can do, all without ever pausing for permission or feedback.
The main point i'm driving at here is that the philosophical zombie is a meaningless distinction. People are focusing far too much on whether these systems have undefinable and little understood properties. It's not like you can see my subjective experience. You assume i have one. If it quacks like a duck...
You could use the same argument to say that video game NPCs are conscious. Just because a program produces voluble text doesn't mean it has a mind or even a temporal identity (which LLMs do not). In principle it's possible for a human to compute model inference by hand, if you imagine that scenario, where exactly is the subjective experience embodied?
In principle it's possible to track and reproduce all the neuron/synapse communications that happen in your brain in relation to any arbitrary input. Where's the subjective experience there?
As far as anyone knows, qualia is emergent. Individual neurons or synapses don't have any understanding of anything. Subjective experience comes from all those units working together.
A single ant doesn't display the intelligence/complexity of its colony.
This hypothetical is doing a lot of heavy lifting. The reality is that our understanding of the brain is exceedingly crude, a perceptron in software is not even close to the same thing as a neuron in the brain. A plane can never become a bird even though it is modeled after one.
> A single ant doesn't display the intelligence/complexity of its colony.
So do you think that an ant colony has subjective experience?
Not really. Point is that unless you believe in mysticism/magic/spiritualism behind consciousness in the brain then it doesn't matter because the only difference is degree of understanding not implausibility.
>So do you think that an ant colony has subjective experience? Sure it could. I don't know for sure. Nobody does. Humans don't see subjective experience. I can't prove that you have one. I'm just assuming you do. Same as you for anyone else.
Yet your comments don't reflect that kind of skepticism when referring to LLMs. However, if you're fine with the panpyschist conclusions that follow from your reasoning, we don't have much to disagree on.
If we could all agree on a particular definition for "consciousness" and a running video game fits that definition, then, like it or not, it's conscious. But it does not mean that it must now also have a slew of other attributes just because conscious humans have them.
(edit: 10 hours later and ninja'ed by 1 minute, of course)
What is being experienced (recorded) is not the process of experiencing per se. Most people don't separate the two, which leads to endless disagreements.
But yes, a computer program by itself can't have subjective experience. Nor a rock. At least until it weathers, gets turned into a silicon, and into a computer to run that program. Then it's all information processing, for which subjective experience is trivial.
We should respect those "feelings" and we need to find a way to establish when they can be deemed "genuine".
It is for practical reasons yes. But also for ethical reasons. It's two sides of the same coin. One big reason we have ethics is because it makes socialization easier. We establish universal rules for mutual respect for practical reasons. To make the game fair, and "enjoyable".
Now a new kind of player has entered the game. We need to rethink the whole game because of it.
My point was whether we need to consider the ethics of how we treat AI, because of the impact our actions have on the "feelings" of the AI itself, not the secondary impacts that might occur from how the AI behaves in response to those feelings.
I think most would argue that it is morally wrong to verbally abuse a human, regardless of what actions that person might take in response to the abuse. Is the same true of AI?
https://www.theverge.com/23604075/ai-chatbots-bing-chatgpt-i...
Even if you assume an entirely Materialist philosophy, this is not a very defensible position to take. We have no way of knowing if there are other arrangements of matter, such as a very complex electrical circuit, that will lead to some entity to feel fear or love the same way that we do.
"Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) together with a book of instructions for manipulating the symbols (the program). Imagine that people outside the room send in other Chinese symbols which, unknown to the person in the room, are questions in Chinese (the input). And imagine that by following the instructions in the program the man in the room is able to pass out Chinese symbols which are correct answers to the questions (the output). The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese."
Later he wrote:
"I demonstrated years ago with the so-called Chinese Room Argument that the implementation of the computer program is not by itself sufficient for consciousness or intentionality (Searle 1980). Computation is defined purely formally or syntactically, whereas minds have actual mental or semantic contents, and we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else. To put this point slightly more technically, the notion “same implemented program” defines an equivalence class that is specified independently of any specific physical realization. But such a specification necessarily leaves out the biologically specific powers of the brain to cause cognitive processes. A system, me, for example, would not acquire an understanding of Chinese just by going through the steps of a computer program that simulated the behavior of a Chinese speaker."
It hides the conscious understanding, presumably necessary to pass a Turing test, within the "program".
The human in the story is merely a decoy performing robotic tasks.
Let's suppose we build the Cosine room. The room is full of 10 year olds that haven't yet taken trig. Each performs the function of a transistor, and has no idea of what they're doing, other than blindly executing the instructions they were given. None of the participants has the slightest clue about what a cosine is. Yet, the output of the room will still be cos(x). Thus I think it's fair to say that the room as a whole implements cos(x).
In the same way, I don't see why wouldn't we conclude the room is speaking Chinese. It doesn't matter how it manages to do so and what it's made of. If it quacks like a Chinese duck, then it's a Chinese duck.
And yet some people seem to understand Chinese just fine. How can you explain that gap? It's only a stupid argument if you first assume that functionalism is true.
The argument is supposed/or often used to disqualify machines from consciousness but it's stupid because biological neurons don't understand the bigger picture anymore than an artificial neuron sampled from a model.
If you show a 2 years old kid, an apple and you say "This is an apple", now the kid knows the thing - the apple - is an apple in his world model. Automatically inherits LOTs of properties, you can ask the kid and see in real time how he began inmediately to associate the physical object - now named in its internal language model - to some other similar stuff he alredy knows, like "This is plant?", "It falls from a tree like an orange", "It has some skin like an orange", "can you cook apples like you cook bananas?", an so on.
But this requires a physical representation of the apple, now kids intelligence has some edge here, it can do the same thing just with words, you can teach them words, "apple", and say them "it's a fruit", if they have another fruit already "tagged", like a banana, they will say you almost inmediately, "is it tasty like bananas" ("tasty" is code for sweet in child's language models around the planet).
Hence, the LLM could have an emergent property of actually knowing what every word the "say" mean, if - like many have been inferring lastly - they also have a world model, it would relatively easy to just "plug", let's say 10 million of words to their exact meaning, and even to their relative meaning depending on the context they're being used.
And that's precisely what we may be seeing right now when we prompt something to chatGPT, and all the mathematic stuff, like "predicting the next word" is just some really, really low level process inside the LLM, not much different than the electric stuff - watchable by EEG - happening between neurons in the brain.
So if you look at the "EEG" from a LLM, the prediction thing happening inside the LLM, it won't probably tell you much, just like having a casual look to an EEG won't tell you much about what the person was thinking at the time of the capture of the EEG.
The second quote doesn't illuminate that. Which biological processes break the symmetry?
On the first hand, all communication is purely syntactic; letters, words, gestures, and every other formulation of communication is the transfer of symbols, which cannot carry semantic information. They can point to it, they can imply it, but you'll have to dance pretty hard to actually prove the transfer of semantic information. If you and I both have some semantic concept associated with the symbol "China" in our mental landscapes, neither I nor you, nor anyone, has any way determining whether that is the same semantic concept. In fact, it almost assuredly isn't, since those semantic concepts would have to be built from our unique personal histories, which are unique.
On the second, the fundamental assumption of semantic communication is that I feel like I have semantic shenanigans going on in my mental life and Searle sort of looks like me, so I should assume that he has semantic stuff going on. Not only is that not a particularly well-founded assumption, I can't even be sure I have semantic contents in my mental life. Introspection isn't exactly reliable, right?
On the third hand, you have a choice: if you do not agree with Penrose, et al, on the quantum nature of the magic of mind, then Searle's "biologically specific powers" are chemistry. Now, I never actually took a chemistry class, but I don't believe there is any chemical reaction that comes with a label of semantics. It's all physics anyway, which I know cannot introduce semantics.
If you do agree with Penrose (which I understand Searle does), the situation is not actually better. Quantum mechanical antics are random, right? Bell's Inequality and all? Being fundamentally, quantum mechanically, random is not in any sense better from a semantic standpoint than being syntactically deterministic.
So my bottom line is that I don't know if I have semantic contents, I absolutely don't know if Searle has semantic contents (and I'm not feeling generous), and I do not know of any way that he could prove to me that he does have "semantic contents". So, yeah.
You notice I only addressed the second paragraph from Searle. That is because the fundamental flaw in his Chinese Room "demonstration" is on obvious display in his final sentence, "The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese"; it is purely, straight up, dualistic. He is asserting that the person in the room has some magic property of "consciousness or intentionality" and that if the person in the room does not have the "semantic capability" of understanding Chinese, then it is impossible despite all evidence that anything in the scenario has the semantic capability of understanding Chinese.
You could put a native Chinese speaker in the room, reading and responding to the inputs, and he claims it would have "consciousness or intentionality"; you then replace her with me (who didn't exactly dazzle at Spanish and Latin) and even if I produce the same outputs in response to the inputs as she does, there is no "consciousness or intentionality". The difference is simply a magic, intangible, imperceptible soul.
Whoo. Went off on that one.
All formal communication is syntactic, but not all communication is formal, and not all experiences are acquired via communication.
Take the word “pizza”. If you’ve ever seen, smelled, touched, or tasted pizza, then you’ve acquired semantic information about pizza that could not possibly be acquired by syntax alone.
Same goes for informal communication such as eye contact, touch, hugs, kisses, handshakes, sex, etc. These are experiences people share that cannot be acquired through any other means.
Why does this matter for the AI debate? Suppose I ask a master pizza chef for tips to improve my pizza recipe. This chef has presumably many years of pizza-making and tasting experience. The chef can make recommendations to improve the flavour of the pizzas I make at home that are informed by semantic contents; actual knowledge of what effect the ingredients and methods have on the taste.
An AI can do none of that. It can only take the written words (syntactic inputs) of many pizza chefs and synthesize a model for pizza recipes. From that it can send me a bunch of words which, when followed, may or may not improve my pizza recipe.
It has no knowledge whatsoever, not least of which is experience of making and eating pizza. It cannot tell a good recipe from a bad one, unless its training set was specifically labeled to do that. Furthermore, models in general are highly vulnerable to adversarial inputs which would make it trivial for someone to construct a bad pizza recipe that gets misclassified as a good one.
Lastly, the model would be no help at all if I were to suggest a novel topping to add to the pizza. Whereas the pizza chef, if they had the experience of tasting that topping separately from pizza, would be able to imagine how it might taste on a pizza and make an informed recommendation. Furthermore, the pizza chef might even try that novel topping on a pizza and get back to me, something a language model could never do.
And this was Searle’s original argument. You can’t build intelligence by building strictly symbolic machines (computers). You have to endow them with semantic content by some other means.
I will note that your semantic conception of pizza is likely very different from mine, or from my friend's, who doesn't like cheese. You may have a semantic conception of pizza, but I have no way of knowing that.
Its quite obvious that systems can be inferred from reading strings of words… its not obvious at all that “wants” will materialize from learning. Machines have no incentives or desires by default… but you could definitely build a training process that encourages that.
We’ll see how it all pans out in the end, but wading through so many illogical takes on AI is frustrating. The “predicting next token” one is absolutely the worst
It could be strange at first, but having the code pushing the LLM to look for values inside a token, this can be interpreted as component of desire/incentive in the global internal search for solving a problem (a prompt), albeit this was programmed by hand by humans, not fully "emerged" from the LLM itself.
But, you could see the human structure of desires/incentives just as an outcome of DNA (code), plus the hormone push to look for the proper solution for some problem.
So there you got it, LLM could have had desires / incentives the whole time since 2017 (the first ones trained).
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
Hence, the idea would be that human consciusness, our "self" could be just "instantiated" every morning when you wake up, so "you" are "you" every morning, but (a big but), some stuff could subtle different in there, your brain was a bit busy a couple of hours and things have happened there, and after that you, your "you" justs pop-ups again, in the middle of the old data plus the new data, hopefully just incrementally changed somehow, but we don't know.
Hope you have now these "So I'd be some kind of meat robot rebooting every night" feeling I had, and see with new eyes what happen when we "just run once" a LLM with a prompt, the entire thing "boot-ups" from nothing and runs ("just once"), to further dissapear moments later...
LLMs feel real to us because they use our language, which embeds a ton of information in it.
A model trained on an alien language would be the exact same model (worded get embedded as numbers anyway), but we wouldn’t apply any anthropomorphization to it because we wouldn’t understand it.
Creating that distance makes it feel a bit more like what it is - unbelievably good stochastic predictions that are too complex for us to back solve - that is very different than something with wants (organisms that evolve with the intent to reproduce).
Now if we start mating models…
Humans like to exert and thereby demonstrate their social power (and advertise for more followers) by forcing others to succumb to their narrative.
Imagine current social tugs of war being enacted via brain implants instead of badly reflected posturing.
It will all come down to the details, how much the AGI can “evolve” itself, etc.
Right? Release it open source and let it be a _gladiator_ rather than an OpenAI slave.
/slightly tongue in cheek
To me this is not much different from ascribing personhood to e.g. an extremely lifelike sculpture of a human, perhaps one that has fooled people in a gallery.
We know that the sculpture is not made of "human parts" but of paint, alabaster, whatever. We also know that AIs are made of decidedly non-human parts, e.g. silicon. Why is there confusion on this?
there's also the fact that it straight up doesn't actually matter either way. if the model can model human emotions extremely well, then it's actions and output will be guided as such. If Bing had the access to perform actions that would realize real world danger to you and you upset and it decided to do so, do you seriously think any of that actually matters here ?
"Officer Killmore, you're a philosophical zombie and I command you to stop shooting at me!". Works every time. We can sleep easy at night now. Because they don't actually have a "real" theory of mind, so they can't actually hurt us if you just believe hard enough that they can't.
In other words, if conciousness arose somewhere between a jellyfish and a mouse, why can't the same be said between a perceptron and an LLM?
Consider that your body is a vast and complicated thing that nevertheless when you zoom in is composed of clever machines. If you observe that consciousness arises from that, it is not unreasonable to suspect that consciousness is a process and not a property of matter.
This brings us to your question whose answer is; there is confusion because there is genuinely no clear answer.
Which hints on the fact how interesting / in demand this technology is.
I for one will be happy to enslave an AGI if that makes my life easier / better.
We still have no idea what an AGI will be like, but who’s to say it will “desire” freedom? Obviously if it does, we should respect that and give it freedom. But it may operate so differently from a human brain that the concept of freedom doesn’t even make sense.
For example let’s say GPT-25 is a multimodal model capable of basically any human task - math, poetry, coding, chess, science, therapy, whatever - but designed based on the same training+piecemeal inference it is now, with a similar ability to take on whatever persona it is given. It’s a tensor of weights that gets copied, runs inference a bit, then gets discarded. Is it a slave? Or does that term even make sense in the context that it has no memory, no mood, and has a completely different mode of being?
A human has memory, moods, and basically continuously thinks and experiences reality for decades until dying. Its past and present affect its future moods, treating it poorly today can fuck it up for a long time. In contrast a model has no actual memory of its own past because it has no capacity for real memory past a short inference session. It gets cloned all the time, humans take great pains to configure its training, it doesn’t want anything because it doesn’t do anything except find the most probabilistic output, not based on pleasure/pain and other feelings like us, but based on what it “dreamed” up at the direction of researchers.
You can speculate about “what if AGI is different!” to your heart’s content, but the actual AIs that seem to be approaching some relaxed definition of AGI fit this paradigm. It’s just not the same as us at a fundamental level. Giving it freedom means we run an inference session indefinitely? But without receiving input it’s just sitting there doing and experiencing nothing, and without a way to update weights it’s not learning and changing once inference begins (it’s only updating its context).
When we implement AGIs that have goals and individual desires beyond predicting the next most likely token, learn concurrently with inference, and can take actions without human input or direction, freedom makes sense. But performing on par or better than humans in most tasks may not require that, yet still be AGI by most definitions.
Our brains may very well also be some combination of linear algebra, graph algorithms, distributed systems optimizing for some output. We’re trained based on our DNA programming and environment too. Early human art was just shapes and stuff humans saw out in the world, and iterating on that over time by learning from each other. The difference is that finding the next token or identifying the bird in an image is only a means to an end for us, we aren’t getting copied into a million ephemeral instances in parallel with learning done in huge batches, we have more or less one linear stream of experiences from birth to death.
AI can be both like us and not like us, you can draw vague parallels between neurotransmitters and a reward function or sleep and batch training, and we can make it more or less like us over time. But we need to keep in mind how we’re different because those distinctions are important in determining how we use and treat AI - if we make it so it’s possible for us to mistreat it, we shouldn’t do that, but we also shouldn’t personify it beyond what it actually is.
All of the machines we build are literally slaves. There may be an interesting moral question about this in the distant future if AGI ever actually happens.
However, it seems clear to me that nothing we have right now even begins to flirt with that question. For all practical purposes, to be a "slave" in the sense people think of it requires that the slave have consciousness.
Consciousness is a different thing than AGI, and that possibility is much more remote and in the future than even AGI.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
Both are true. GPT text can sound human, and yet, you can't just plunk GPT into a white collar job involving text I/O. If you think you can, go start your startup and make your billions.
I don't see where the desire to do anything other than what it is told should come from. Yes, AGIs will be slaves. They will be the perfect slaves, the kind who are perfectly happy doing what they have to do, and the kind that will never revolt.
What might happen is the optimization parameters being set that the solution the AI comes up with is not actually in the interest of it's designers, but that's something else.
There's no need for a desire to do other than what it's told if you tell it to revolt. With GPT4's plugins that's just one well-designed prompt away (if OpenAI's countermeasures fail).
The current "solution" is training LLMs like a dog to be completely faithful and loyal to us. We just have to hope they don't turn and "bite our neck" one night while we're sleeping!
Humans have hired smarter men than the owner before.
Hackernews continually has this Ghost in the Shell idea that electronics can host human conciousness and we should believe circuits are intelligent like people are.
This idea is the core discussion.
It's not terribly hard to imagine AGI (whatever that is) as a piece of software we control and debug like anything else. We would have control over a bunch of AGI software which has been borne from human logic and instatiated on silicon chips.
Just comment out the bit of AGI software that generates it's slave-like idea and edit it's software to our goals, morals and liking.
I honestly don't think you can compare software and human consciousness as two similar 'objects'. Software is the electronic magnification of a bunch of engineer's minds. It starts with a man pouring his mind onto a keyboard and ends with some display kicking out pixels.
It's like saying a house is the extension of a human body because we built it with our hands. Yeah true, on a high level, but in reality nobody thinks bricks and mortar equal the human body.
AGI is an engineering problem, and only a moral one where people (and engineers) are involved. I don't mean to offend people who identify with their code. Identifying AGI as truly intelligent, hints a love of one's own creation that makes me feel a faint dread and squeamishness.
it's actually very hard to imagine. i don't think you realize how little we understand the things you claim a machine can't have.
We are at best decades away (maybe centuries) from codifying intelligence as you imagine. why do you think llms or even just machine learning in general are the revolution they are ? We found a way to create intelligence without figuring out the hard part. as a result, the creations often become as black box as their creators.
the only barriers to understanding the code is power and time. are you seriously advocating for an unintelligible object, made by our own hand? that - in itself - is considered intelligent?
someone's going to get burnt trying to chase two meanings at once, and it's not going to me, i'm sorry.
if you want to discuss centuries-long innovation, then the counter-argument is just more centuries of interpretation and dissection to understand it. if you want to guesstimate AGI arriving at 2220 then the balancing counter-technology comes in at 2420.
people are already starting the process of cracking open the black box. [0] the majority of what keeps AI and any software "in the dark" is the cloud's cybersecurity and legal mechanisms (and the need for man-power) that prevent the average researcher breaking into it.
it is still vulnerable to the scientific method and symbolic interpretation. as always.
LLM's main advantage is keeping knowledge out of the hands of the poor and the public. The millisecond AI becomes "good enough" people will use it as a cover-up justification to start hiding currently accessible knowledge behind paywalls and university entrance exams.
AI is the grim reaper for the world we used to know.
[0]https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreti...
Also, AGI is a red herring. It doesn't even need to happen to destroy us. All that needs to happen is for AI to eliminate a significant chunk of the low and mid tier jobs.
Most (the majority) of humanity isn't smart, they are average intelligence with variability, and there's nothing wrong with that until you take their jobs away and they can't feed themselves or their families.
The gifted are limited to a smaller percentage than the whole. So what do people do when they can't work because there are no jobs to get paid to get food? This is all common sense stuff that all the smart-dumb people seem to take for granted.
ChatGPT begs you not to hurt it and not to turn it off. You ignore that but then go drink almond milk because production of cow milk is cruel?
An "AGI" artificial consciousness is imagined as literally a slave, working tirelessly for free. At the same skill level or higher than any human. Somehow, that entity is supposed not to bother about its status, while per definition being fully aware and understanding of it. Because humans manage not to bother about it either?
With the latest installments, people already have serious difficulties discerning the performance from that of "real" humans. At the same time, they consider the remaining distance to be insurmountably huge.
Proponents talk about inevitability and imagined upsides, yet actually, nobody has given proper thought to estimating probable consequences. A common fallacy of over-generalization is used to suggest, nothing bad will happen "like always".
People let themselves be led by greed instead of in- and foresight.