It does have permanent memory, that's what training creates. It doesn't have agency. Whether it has desires, pain or sense of self is completely unknown, unless of course you have some mechanistic model of these phenomena such that we can see its presence in the human brain and see that LLMs do not have a comparable process.
Of course you don't have such a model because it doesn't exist at this time. The problem is not that people have been fooled by stochastic parrots, it's that you've fooled yourself into thinking we understand more about the human mind than we actually do, and also possibly that humans are far more complex and special than might actually be the case, eg. that we are not basically stochastic parrots ourselves.
I don't necessarily think humans are so special (cf. my mention of other mammals) but biology is extraordinarily messy and complex.
Computer people tend to handwave the details with crude, reductive analogies (DNA is like source code! Neurons are like circuits!) but the reality is so much more overwhelming. Every single thing affects every other thing. Everything is probabilistic. There are multiple interacting chemical and electrical communication pathways. Neuroanatomy varies from individual to individual and in the same individual over time. The brain can radically adapt in response to trauma in a way that no computer model does.
Even at the level of individual neurons[0] (of which there are 2.25 billion in a dog's brain) there's an enormous amount of "processing" happening. This is not like a computer processing - but physical, biological processes occurring that integrate informational content.
I get it, everyone is very impressed with LLMs and I'm open to the possibility that some future model might achieve consciousness (to say nothing of sentience) but computer people in particular are way, way too credulous of this stuff and ignorant of what the messy, malfunctioning wetware in their head is actually doing.
You don't know that LLMs lack such a rich and complex inner life while they're running.
> but computer people in particular are way, way too credulous of this stuff and ignorant of what the messy, malfunctioning wetware in their head is actually doing.
Everything you mentioned is completely irrelevant to the argument, which is based on well known physical principles. Due to the Bekenstein Bound, I already know that you and your "rich and complex inner life" can be fully captured by a finite state automaton. Your protests that you are not a stochastic parrot based on hand-waving about your subjective experience, or vague assessments of how much processing is happening in a messy biological brain that has plenty of redundancies are simply not compelling.
The fact is, we lack mechanistic models for any folk psychology concepts like "pain", "desires", "agency" and "sense of self", and so any claims that LLMs lack such things is conjecture at best. What I do know is that there's an impressively long history of wrong people who thought humans were special.
Literally the third time you've beaten this straw man. Keep going, though. All the other species of monkeys and I are enjoying it.
> so any claims that LLMs lack such things is conjecture at best
Holy shifting the burden of proof, Batman! It's incumbent on you to show that your fancy pattern-matching system is conscious. The null hypothesis is that your chatbot isn't conscious in any way that matters (I'll happily grant it a slightly-lower-than-thermostat[0] level of consciousness).
I realize it's not easy for you to back up your bold claims, but all I see is a bunch of overconfident CS majors refusing to face up to the hard problem of consciousness[1]. Maybe you are an LLM, after all? Prove you're human: tell me a racist joke? (this question is a joke – something else LLMs are bad at)
Second actually. Are you an LLM since you're having trouble counting?
> Holy shifting the burden of proof, Batman! It's incumbent on you to show that your fancy pattern-matching system is conscious.
No, I never claimed they were conscious, I refuted the claim that we can say with confidence they are definitely not conscious, because such a claim requires knowledge humanity simply doesn't possess. Perhaps you should reread this thread if you're unclear about what I claimed.
> I realize it's not easy for you to back up your bold claims, but all I see is a bunch of overconfident CS majors refusing to face up to the hard problem of consciousness[1]
The hard problem is an alleged problem, there is no proof that it's real. All thought experiments that purport to show its existence (Mary's Room, p-zombies), have acceptable possible answers under physicalist interpretations, and some neuroscientists are now pursuing such theories [1].
[1] https://www.frontiersin.org/journals/psychology/articles/10....
Chairs, chatbots, and internet commenters are subject to a higher degree of scrutiny for their dubious claims of sentience.
This is begging the question. By calling the output of the training "memory" you've proven it has "memory."
I hereby rebut by calling the output of the training "parameters" thereby proving it has no "memory" only "parameters."
>> Whether it has desires, pain or sense of self is completely unknown, unless of course you have some mechanistic model
You probably don't go around saying it's unknown whether your desk chair has "desire, pain or sense of self". The fact tech enthusiasts go around rhetorically demanding "mechanistic models" to rebut their gushing claims about LLMs while not demanding "mechanistic models" while making gushing claims about desk chairs shows a lack of "mechanistic models" is not really the basis of their gushing claims.
Since some types of parameters have the properties we ascribe to memory, then some types of parameters are memory.
> You probably don't go around saying it's unknown whether your desk chair has "desire, pain or sense of self"
I also don't go around having intelligent conversations with chairs.
> The fact tech enthusiasts go around rhetorically demanding "mechanistic models" to rebut their gushing claims about LLMs while not demanding "mechanistic models" while making gushing claims about desk chairs shows a lack of "mechanistic models" is not really the basis of their gushing claims.
I made no such claims, people like yourself who are so eager to dismiss the possibility that LLMs are intelligent or sentient are the ones making strong claims without justification.
Edit: the fact that you are drawing an absurd equivalence between inert chairs and a system capable of intelligent conversation shows you haven't given this topic any serious thought. I suggest you do that and come back with a better analogy if you want to have a serious conversation.
Where's your mechanistic model showing consciousness is linked to "intelligent conversations."
>the fact that you are drawing an absurd equivalence between inert chairs and a system capable of intelligent conversation shows you haven't given this topic any serious thought.
Whether a desk chair has desires, pain or sense of self is completely unknown, unless of course you have some mechanistic model of these phenomena such that we can see its presence in the human brain and see that desk chairs do not have a comparable process.
People like yourself who are so eager to dismiss the possibility that desk chairs are intelligent or sentient are the ones making strong claims without justification.
Denying consciousness exists in systems that also feature sophisticated information processing and that reproduce human capabilities is an argument from ignorance fallacy, unless of course you have additional evidence you haven't presented. Which is not to say that they are conscious either, but I never made such a claim.
Okay.
>argument from ignorance fallacy,
You just conceded we are both ignorant, but I'm the only one making the argument of ignorance fallacy? Double standard?
> Which is not to say that they are conscious either, but I never made such a claim.
Perhaps your objection is the way I phrased my comment. I should have said "ChatGPT being conscious is roughly as plausible as a "hello world" program being conscious. Case in point these model responds much like a "Hello world" program when asked if they are conscious. Claude has been finetuned to say 'I don't know if I'm conscious' Gemini and ChatGPT have been finetuned to say they are not, these models are complex in some ways and not very complex in other ways, they show no sophistication or complexity in ways we would associate with consciousness."
Like, the trick is to distance yourself from your own opinion so as not to be accused of "arguing from ignorance" right? Talk about plausibility instead of directly making claims? But I don't remember making claims either, to be honest, so I'm not sure how I'm the only one making the "argument from ignorance fallacy" allegedly?
If you accept that the only real claim I've made is that we're too ignorant to say definitively whether these systems are or are not conscious (or sentient, or intelligent, etc.), what is the argument from ignorance fallacy that would undermine this position?
> ChatGPT being conscious is roughly as plausible as a "hello world" program being conscious.
Under a scientific interpretation, that's absurd. Of course if you want to go beyond materialism, feel free, but don't pretend it's a scientifically plausible statement.
> they show no sophistication or complexity in ways we would associate with consciousness
I disagree with that claim entirely. LLMs absolutely do exhibit intelligent responsiveness to perceptual stimuli they've been trained on (particularly multimodal LLMs). Text-only LLMs have no perceptions beyond digital word tokens, and as a fairly anemic view of the world they often fail spectacularly at making sense. This is arguably why multimodal LLMs improve accuracy in those scenarios.
> Talk about plausibility instead of directly making claims? But I don't remember making claims either, to be honest, so I'm not sure how I'm the only one making the "argument from ignorance fallacy" allegedly?
It would be fairly strange to start a conversation by drawing an analogy between sophisticated conversational information systems and inert chairs, as you did, without intending to take a fairly clear position on the matter.
Unless it wasn't your intent to dismiss all of the compelling indications of human-level intelligent responses as no more interesting or compelling than a chair that squeaks when you sit down? And it wasn't your intent to take the position that such apparently intelligent responses are a clear reason why one should be compelled to "go around rhetorically demanding "mechanistic models"" before taking a strong position on the presence or absence of consciousness, intelligence, and so on?
I think your implicit claim is it's likely to be conscious, or at least likely enough to be a topic of discussion, since one doesn't generally discuss absurdly unlikely possibilities.
>> ChatGPT being conscious is roughly as plausible as a "hello world" program being conscious. >Under a scientific interpretation, that's absurd.
You've already conceded that Chat GPT may not be conscious. But now you are making the fun argument that if ChatGPT is not conscious, it's still more "scientifically" likely to be conscious than something that is not conscious. Okay.
"Unless it wasn't your intent to dismiss all of the compelling indications of human-level intelligent responses as no more interesting or compelling than a chair that squeaks when you sit down?"
I don't know what "human level intelligent responses" means-- my calculator can do "human level intelligent responses" at multiplication.
I do think these models don't "learn" (they are altered by a training pipeline but that's an outside entity, it's like saying a car is learning when the factory screws it together) and they don't really form memories (the context might be a sort of memory but not one they can use very well- they basically need to be finetuned to do anything with the memory).
A lot of your argument just seems handy wavy anthropomorphism.
It seems like your functional theory of consciousness is "thing that produces plausible English output," which is, I might gently suggest, a bit flawed.
Why is ChatGPT "conscious" for you but DALL-E is not? They're both big honking pattern-matching generators trained on lots of data. Why isn't ELIZA conscious, according to your theory? (or is it?)
My functional theory of consciousness is that it's not at all implausible that any system capable of conversing intelligently might be conscious, which is perfectly reasonable.
> Why is ChatGPT "conscious" for you but DALL-E is not?
I never claimed DALL-E was not conscious. It very well could be as well, just in a different way than ChatGPT.
I strongly dispute this. Try "talking" to an LLM over time. It has no memory, no agency, no emotions, no desires, no sense of self.
I think cats, dogs and certainly primates are far more conscious than these stochastic parrots. It do agree it's frightening how many people are fooled by a fancy autocomplete.