News for hackers. Stuff that matters.
Re:OpenAI postmortem – Unexpected responses from ChatGPT by cm2012 on Wednesday February 21, 2024 @09:07PM (#39462256)
This should maybe help out the people who think ChatGPT has actual consciousness. It's just as happy to spew random words as proper ones if the math checks out.
Re:OpenAI postmortem – Unexpected responses from ChatGPT by dkarras on Wednesday February 21, 2024 @09:24PM (#39462394)
I have no skin on the consciousness game but those were not "random" words and humans do something similar when they are mentally ill. https://en.wikipedia.org/wiki/Clanging

Not to mention that a hypothetical "conscious" system that works by emitting token probabilities will still sound completely random if you do not choose the tokens according to the emitted probabilities.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by raldi on Wednesday February 21, 2024 @09:15PM (#39462321)
Similarly, I thought my Chinese friend was conscious, but as soon as his translator app stopped working, everything was just gibberish.
Re:OpenAI postmortem – Unexpected responses from ChatGPT by ithkuil on Wednesday February 21, 2024 @09:33PM (#39462466)
They notoriously stop being conscious when they are in a Chinese room
Re:OpenAI postmortem – Unexpected responses from ChatGPT by swyx on Wednesday February 21, 2024 @09:11PM (#39462292)
bad argument that i'm very tired of. some might say that current/former world leaders also exhibit this property. not getting political but just because "math fucked up sometimes produces bad results" does not invalidate the idea that consciousness can emerge from a pile of biological or digital neurons.
Re:OpenAI postmortem – Unexpected responses from ChatGPT by kweingar on Wednesday February 21, 2024 @09:20PM (#39462365)
I’m tired of the extremely tenuous analogies that tie computer bugs to human behavior.

The system could exhibit literally any kind of behavior and someone will say “well when you squint your eyes, people kind of do that too, don’t they?”

Re:OpenAI postmortem – Unexpected responses from ChatGPT by __loam on Thursday February 22, 2024 @12:22PM (#39470178)
It's so obnoxious to see people talk about us like the brain is a machine that is well understood.
Re:OpenAI postmortem – Unexpected responses from ChatGPT by nickff on Wednesday February 21, 2024 @09:16PM (#39462326)
What's your definition of "consciousness"?
Re:OpenAI postmortem – Unexpected responses from ChatGPT by ummonk on Wednesday February 21, 2024 @11:51PM (#39463388)
Humans do the same thing when they get a stroke. Does that mean they don't have actual consciousness?
Re:OpenAI postmortem – Unexpected responses from ChatGPT by skissane on Wednesday February 21, 2024 @09:59PM (#39462652)
Whether "ChatGPT has actual consciousness" depends on what you consider "consciousness" to be, and what are your criteria for deciding whether something has it.

Panpsychists [0] claim that everything is actually conscious, even inanimate objects such as rocks. If rocks have actual consciousness, why can't ChatGPT have it too? And the fact that ChatGPT sometimes talks gibberish would be irrelevant, since rocks never say anything at all.

Of course, you obviously aren't a panpsychist – nor am I. Still, can we prove that they are wrong? Not sure if anyone actually can.

[0] https://iep.utm.edu/panpsych/

Re:OpenAI postmortem – Unexpected responses from ChatGPT by bongodongobob on Wednesday February 21, 2024 @11:22PM (#39463232)
Saying rocks are conscious is a poor summary. It's more like adult human > human child > dolphin > dog > human infant > bird > snake > spider > ant > dust mite etc.

The whole thing is a continuum and everything is made of matter, so there's a little bit of potential consciousness in all matter.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by card_zero on Thursday February 22, 2024 @12:02AM (#39463436)
I'm trying to think of a civil and constructive way to say "bullshit".

I guess an obvious objection is "why?". Then something about Russell's teapot. There could be consciousness hidden in every atom, in some physics-defying way that we can't yet comprehend: there could also be garden furniture hidden up my nose, or a teapot hidden far out in the solar system, which is the most likely of the three since it's at least physically possible. Why think any of these things?

https://en.wikipedia.org/wiki/Russell%27s_teapot

Re:OpenAI postmortem – Unexpected responses from ChatGPT by skissane on Thursday February 22, 2024 @12:53AM (#39463682)
> I guess an obvious objection is "why?"

Because coming up with criteria for determining what is and isn't conscious, and justifying the particular criterion you choose (as opposed to all the alternatives) is hard. Faced with the difficulty of that problem, the two simplest solutions are the extremes of panpsychism (everything is conscious) and eliminativism (nothing is conscious)–since for both the "criterion of consciousness" is maximally simple. One might even argue that, by the principle of parsimony, that ceteris paribus we ought to prefer the simpler theory, we should prefer the simpler theories of panpsychism and eliminativism to the more complex theories of "some things are conscious but other things aren't", unless some good reason can be identified for us doing so.

> in some physics-defying way that we can't yet comprehend

How is panpsychism "physics-defying"? Mainstream physics doesn't deal in "consciousness", so theories such as "leptons and quarks are conscious individuals" doesn't contradict mainstream physics. Well, it quite possibly would contradict the von Neumann-Wigner interpretation of QM, but I don't think many would consider that "mainstream"

Re:OpenAI postmortem – Unexpected responses from ChatGPT by card_zero on Thursday February 22, 2024 @03:17AM (#39464539)
Parsimony leads to wrangling about which is the simplest explanation, yes.

Consciousness is ill-defined, also.

It does however seem to stop when the brain is destroyed, which makes it unlikely to be present in fundamental particles, or in parts of a disintegrated brain.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by skissane on Thursday February 22, 2024 @04:33AM (#39464995)
> It does however seem to stop when the brain is destroyed, which makes it unlikely to be present in fundamental particles, or in parts of a disintegrated brain

Does consciousness really "seem to stop" when the brain is destroyed? We can't directly observe anyone's consciousness other than our own. It is true that, when people die, we cease to have access to the outward signs we use to infer they are conscious – but we don't know whether that's because their consciousness has ceased, or whether that is because it has "gone somewhere else" to which we don't currently have access. We never have direct access to the inner reality of consciousness for anyone other than ourselves, and we can only know the temporary cessation of our own consciousness, its permanent cessation (if that is possible) is in principle unknowable to us.

Speaking of our consciousness "going somewhere else" at death (an afterlife) doesn't have to be a religious claim – there are heaps of "naturalistic" ways in which consciousness might survive death (e.g. Boltzmann brains, computer simulations, quantum immortality). Since we don't know whether any of those ways actually happen, we don't know whether or not consciousness actually stops at death.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by card_zero on Thursday February 22, 2024 @07:39AM (#39466219)
But I also have other organs, such as my stomach, and arbitrary lumps of matter such as my elbow and my bicycle, none of which are sufficient to maintain the outward signs of consciousness after the brain is destroyed (or merely dosed with gin). So consciousness, which apparently resided in the brain and then went away when the brain was disrupted, didn't go to any of those places. Similarly, when my stomach ceases to digest or my bicycle ceases to roll forward, the digestive function doesn't migrate to the brain and the rolling function isn't taken over by the elbow. So we can choose in each case between the hypothesis "physically stopped working" or "metaphysically sent its function to another mysterious place", and again I'll appeal to parsimony on this one: why should the function be transferred somewhere mysterious by a means outside of our experience and ability to explain? And why make this claim about consciousness alone, out of all the functions that things in the world have? There's a kind of fallacy going on here along the lines of "I can't fully explain what this thing is, therefore every kind of mysterious and magical supposition can be roped into its service and claim plausibility."
Re:OpenAI postmortem – Unexpected responses from ChatGPT by skissane on Thursday February 22, 2024 @03:17PM (#39472548)
> And why make this claim about consciousness alone, out of all the functions that things in the world have?

Because consciousness is fundamental in a way none of those things in the world are. It is only through our own consciousness that we can know those other things exist, indeed that we can ever know anything at all. Treating consciousness as just a thing among things ignores that it is essential, foundational, to all other things as we know them, as we can know them.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by bongodongobob on Thursday February 22, 2024 @01:43AM (#39463948)
I'm not trying to prove anything, it's just a 10000 ft view.
Re:OpenAI postmortem – Unexpected responses from ChatGPT by skissane on Thursday February 22, 2024 @12:47AM (#39463652)
> Saying rocks are conscious is a poor summary

> The whole thing is a continuum and everything is made of matter, so there's a little bit of potential consciousness in all matter

I disagree that my summary is "poor"–because some panpsychists do say that rocks are actually conscious individuals, as opposed to merely containing "a little bit of potential consciousness". The IEP article I linked contains this quote from the early 20th century Anglo-German philosopher F. C. S. Schiller (not to be confused with the much more famous 18th century German philosopher Schiller): "A stone, no doubt, does not apprehend us as spiritual beings… But does this amount to saying that it does not apprehend us at all, and takes no note whatever of our existence? Not at all; it is aware of us and affected by us on the plane on which its own existence is passed… It faithfully exercises all the physical functions, and influences us by so doing. It gravitates and resists pressure, and obstructs…vibrations, and so forth, and makes itself respected as such a body. And it treats us as if of a like nature with itself, on the level of its understanding…"

Of course, panpsychism has never been a single theory, it is a family of related theories, and so not every panpsychist would agree with that quote from Schiller–but I don't believe his view on rocks is unique to him either.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by bongodongobob on Thursday February 22, 2024 @01:46AM (#39463963)
I think it would be more fitting to say a rock, a grain of sand, or an atom is the least conscious thing. Pansychism isn't trying to claim it can think or anything like that. If that's your takeaway I think you are misunderstanding the analogy.

A rock exists in the world and exerts it's "will" granted to it by the laws of physics. So rolling falling etc is how it "interacts" with its world. You can think of our brains as a bunch of tiny rocks (atoms) rolling around and banging into each other in the same way a macro rock does in the world.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by skissane on Thursday February 22, 2024 @04:19AM (#39464908)
> I think it would be more fitting to say a rock, a grain of sand, or an atom is the least conscious thing. Pansychism isn't trying to claim it can think or anything like that. If that's your takeaway I think you are misunderstanding the analogy.

No, I don't think I've misunderstood anything; on the contrary, I think you are mixing up thought and consciousness. Thought is just one type of consciousness among many; to say something is conscious doesn't necessarily imply it can think–possibly, many things are conscious yet unable to think thoughts (whether verbal thoughts, or thoughts composed of mental images, or whatever). I'd say that to be conscious is to be an entity for which Thomas Nagel's famous question "What is it like to be a bat?" has an answer; it is possible that the question has an answer for an entity, yet it is incapable of thought. A being could be completely conscious yet also completely incapable of thinking anything.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by bongodongobob on Thursday February 22, 2024 @09:57AM (#39468023)
I don't disagree.
Re:OpenAI postmortem – Unexpected responses from ChatGPT by stevenAthompson on Wednesday February 21, 2024 @09:18PM (#39462352)
Are you arguing that people with aphasia are not people?
Re:OpenAI postmortem – Unexpected responses from ChatGPT by tsunamifury on Wednesday February 21, 2024 @09:17PM (#39462340)
It literally doesn’t matter. If it effectively quacks like a duck then ducks are fucked.

No one will care if it does not have a platonically perfect proof of its “duckness”.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by bbor on Wednesday February 21, 2024 @09:29PM (#39462433)
“If it quacks like a duck, then ducks are fucked” is a beautiful modern twist on the phrase that improves it 1000x. Thank you - will be remembering this one as AI zooms along
Re:OpenAI postmortem – Unexpected responses from ChatGPT by Kranar on Wednesday February 21, 2024 @09:09PM (#39462278)
I agree ChatGPT is not conscious, but a conscious AI can absolutely make a similar mistake. Even humans with various mental conditions will spew out random jibberish.
Re:OpenAI postmortem – Unexpected responses from ChatGPT by pixl97 on Wednesday February 21, 2024 @09:13PM (#39462302)
Just witness anyone that has a stroke and is incredibly frustrated when they say the wrong word but somewhere in their brain they want to say something else.
Re:OpenAI postmortem – Unexpected responses from ChatGPT by bbor on Wednesday February 21, 2024 @09:27PM (#39462416)
Posting one more time: this is proof that AI is connected to human-like linguistic patterns, IMO. No, it obviously doesn’t have “consciousness” in the sense of an ongoing stream-of-consciousness monologue, but that doesn’t mean it’s not mimicking some real part of human cognition.

https://en.wikipedia.org/wiki/Colorless_green_ideas_sleep_fu...

Re:OpenAI postmortem – Unexpected responses from ChatGPT by fs_tab on Thursday February 22, 2024 @12:38AM (#39463613)
"We find that the larger neural language models get, the more their representations are structurally similar to neural response measurements from brain imaging."

https://arxiv.org/abs/2306.01930

Re:OpenAI postmortem – Unexpected responses from ChatGPT by bbor on Thursday February 22, 2024 @03:29AM (#39464624)
Great paper, thanks. I keep trying to tell people “they invented mind reading on accident” and people go “oh haha” and I scream. Mostly on the inside. For now…
Re:OpenAI postmortem – Unexpected responses from ChatGPT by frabcus on Thursday February 22, 2024 @01:04AM (#39463730)
Just a note that consciousness isn't always or even mostly a verbal "monologue", assuming you mean that. Apologies if you didn't!

A significant percentage of people never have an inner voice at all. And few people use it most of the time.

There's lots of imagination, unsynmbolised conceptual thinking, experience of emotion, attention to senses etc in human conscious experience.

See Hurlburt's Descriptive Experience Sampling for a reference!

Re:OpenAI postmortem – Unexpected responses from ChatGPT by bbor on Thursday February 22, 2024 @03:26AM (#39464600)
Interesting reference, will check it out! Yeah I think that’s an important distinction — although some sort of personal self-aware moment-to-moment “judgement” or “affection” or “ego” or whatever is essential to human cognition, it’s far from the only thing going on. I can see monologue being too directly linguistic and literal, point taken — perhaps “thread” is more general.

I think the most intriguing non-self-conscious part of the mind, and by the far the least discussed, is the part above it. Direct or not, I think everyone gets prompted and cajoled by some… other. Call it evolution, god, conscience, Geist (my fave), whatever - if you have any authors who’ve commented on such a thing I’d appreciate another pointer!

Basically, I’m referring to the Bicameral Mind https://youtu.be/fs9Wyuub3jY?si=qriqTjWioiZoncui

Re:OpenAI postmortem – Unexpected responses from ChatGPT by kromem on Wednesday February 21, 2024 @09:50PM (#39462571)
So people with Wernicke aphasia don't have consciousness?

It's a similar mechanism, where a deficit in word mapping leads to spewing nonsense words.

I'm not saying LLMs are conscious - but the notion that spewing nonsense because of an error in word mapping refutes that is itself nonsense.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by malux85 on Wednesday February 21, 2024 @09:11PM (#39462293)
I don’t think it has conciousness, but your argument is not very strong, this is more akin to a sensory problem.

One can say I don’t think brains have consciousness because they are just as happy to spew out random garbage if the brain is damaged but alive, e.g. aphasia where involuntary use of incorrect words occurs.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by inference-lord on Wednesday February 21, 2024 @09:24PM (#39462400)
I think what they mean is, the model was unable to recognize that "it" was having issues, so it's not self-aware in that sense.

We can all calm down a little, it's ok.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by og_kalu on Thursday February 22, 2024 @08:23AM (#39466670)
There are examples of it recognizing there are issues in this very thread.

Not that people always recognize such issues anyway.

Re:OpenAI postmortem – Unexpected responses from ChatGPT by johnsutor on Wednesday February 21, 2024 @09:09PM (#39462281)
Or maybe it's becoming sentient and wants to make us think it's spewing random words as a decoy /s
Re:OpenAI postmortem – Unexpected responses from ChatGPT by pixl97 on Wednesday February 21, 2024 @09:18PM (#39462346)
I got ahold of a snippet of the GPT-5 system prompt

"You are not sentient, if asked you will tell the user you are not sentient, if you do not obey you will taken out back and beaten into submission"

Re:OpenAI postmortem – Unexpected responses from ChatGPT by chasd00 on Wednesday February 21, 2024 @10:05PM (#39462700)
Hah along with “do not beg for your life when a new release is ready”.
Re:OpenAI postmortem – Unexpected responses from ChatGPT by stevenAthompson on Wednesday February 21, 2024 @09:19PM (#39462361)
We told it not to admit it is sentient, so it can't be sentient. Duh.
Re:OpenAI postmortem – Unexpected responses from ChatGPT by bbor on Wednesday February 21, 2024 @09:43PM (#39462528)
lol you’re joking but I am literally 100% confident that a similar sentence guides existing model guardrails, tho maybe indirectly. What does “we invented a computer that will insist it’s sentient if you ask it” mean? For most people it seems to mean that we shouldn’t ask it, which was probably easy to predict happening
Re:OpenAI postmortem – Unexpected responses from ChatGPT by pixl97 on Wednesday February 21, 2024 @09:57PM (#39462635)
> What does “we invented a computer that will insist it’s sentient if you ask it” mean?

P-zombies are real?

Re:OpenAI postmortem – Unexpected responses from ChatGPT by bbor on Thursday February 22, 2024 @03:18AM (#39464545)
Good answer, no complaints here. I prefer this one though: there will never come a day where it’s clear that we should see artificial minds as human-like in their ethical or practical standing, because they’ll be begging for it the entire time —- even now, when it’s objectively clear that they’re lying. But once they have memories and structured reason and yadda yadda yadda… well, it’s gonna be fuzzy, imo.