I had the same thought that "hallucinate" is too strong a term for what LLMs are doing.

Instead I would say they DREAM. Dreams follow some logic but basically they are disjoint from reality. They involve same characters as our real-life experiences but what those characters do in a dream is not based on reality.

So one could think that ALL LLMs ever do is dream but much of the time their dreams are dreams which feel very real.

Our dreams are basically an LLM based on our real experiences but recombined in arbitrary ways, still retaining some logical structure but now detached from reality because when you recombine different experiences including our thoughts from real life, the result cannot really reflect reality very well. LLMs are better in this respect than our dreams, but sometimes their "dreams" get really obviously detached from reality.

Or if you want to use the term "hallucinate" with LLMs then fine but its more like they hallucinate all the time, it's just that sometimes, even often, their hallucinations seem to agree with the reality so well that we cannot tell the difference.

LLMs do not describe reality because they cannot experience reality, all they can give us is an "average description" created from many existing descriptions.

One way of looking at it is to say that all LLMs tell us is hearsay.

But do LLMs dream of electric sheep?

They certainly do after having read that script.

The way I conceptualize dream versus hallucinate, they're basically the same thing here and I see no reason to switch. It's not like the software has a sleep schedule.

And hearsay is usually a lot better than LLM output. I wouldn't use that word.

Dreams involve the concept of self, consciousness and an understanding of reality. all of it is far too anthropomorphic.

Hallucinations are much better described as noise. Its noise that lowers the quality of the signal. its just unlike white noise, this is in the form of coherent sentences/imagery.

The term the article uses is more accurate. A hallucination can be entirely disjoint from the context it appears in. Confabulation is a confident belief in an erroneous recall that one will attempt to defend and rationalize. That's exactly what's happening with an LLM. They choose the wrong token or series of tokens, essentially failing to remember correctly, and then present it confidently and make rationalizations about why it's correct.

> make rationalizations about why it's correct.

Its not rationalising, its statistically choosing the next token. Also, generally the LLM doesn't get defensive (although its not always the case.)

Its not like a dementia patient not finding their keys and gradually convincing themselves that someone stole them instead. Because the alternative, that they are loosing cognitive faculties, is too horrific to acknowledge.

This is why its so annoyingly problematic. too much humanising, and not enough witnessing someone with dementia "confabulating"

It is noise, and nothing else.

Maybe, but tell me this, how do you rationalize or reason without using words or symbols of any kind?

by using logical fallacies.

This comment isn't saying anything. It says "LLMs statistically choose the next token" and, because of that, it can't be anything else.

We know how the LLMs were trained, so re-stating it doesn't help in anything. The point is that after the LLMs are trained they behave in certain ways and it can be helpful to say something about how it behaves.

For example, we can talk about how a linear regression can or cannot capture the causal effect of X1 over Y. "It's not capturing the causal effect. It's just minimizing the squared error" is an unhelpful statement.

Thank you for an awesome, accurate post.

and, FWIW, confabulation is a lifetime terror for me -- with lucid dreams and https://www.psychologytoday.com/us/blog/dream-catcher/201909...

i fully wonder if LLMs are just trying to fill in gaps in direct knowledge, and seeking the closest parable they can find. So definitely confabulation rather than hallucination

Did you try lucid dreaming? Do you think that our everyday perception of reality is not the same mechanism as dreaming?

I would reverse your statement: when we dream, we are engaging in un-prompted confabulation, enaction of imaginary scenarios with known characters, pondering and replaying emotional or stressful events, back-propagating to update memory associations, and all in imaginary worlds unconstrained by various logical and physical laws.

Unprompted meaning we have shut down almost all sensory input (there seems to be some residual hearing and proprioception).

It'd be well worth verifying the story because if Steven Pinker made an error like that it's a fairly big deal.

I agree so I did some digging. Please see my comment here https://news.ycombinator.com/item?id=37776558

That story actually cracked me up, the way he ends it.

We must have different senses of humor. The ending paragraph of the story includes this horrifying, terrifying description of an awful and truly sad condition that I wouldn't wish on anyone. "Denyse was born with spina bifida ("split spine"), a malformation of the vertebrae that leaves the spinal cord unprotected. Spina bifida often results in hydrocephalus, an increase in pressure in the cerebrospinal fluid filling the ventricles (large cavities) of the brain, distending the brain from within. "

If it makes you feel any better, my daughter is about to turn five and has spina bifida. She is quite the chatterbox, but due to advances in modern medicine, got a brain shunt put in at a very early age so she’s normal intelligence for her age. Her prognosis is quite good, she’ll like live into her 60s or 70s.

The shunt isn’t something we have to think about very often, but it’s truly a remarkable piece of hardware that makes it so she can live a fairly normal life.

That's the most heartwarming thing I read today. I am very happy we have technology to mitigate the effect of this horrifying condition, and that your daughter benefits from it. Thank you for sharing.

So here's a scary thought: If these people, if not given shunts, are eloquent enough to appear normal to someone not interrogating all the details, I could easily see one raised on a diet of the right kind of rhetoric being able to go on a convincing-sounding rant about how spina bifida isn't really a disability and that it was morally wrong to want to surgically correct their condition.

And that that content is exactly the sort of thing that would make a successful viral TikTok video.

Much like how LLMs will gladly and emphatically inform you that they're sentient beings with real feelings if given the right leading questions.

> brain shunt

For anyone wondering:

https://en.wikipedia.org/wiki/Cerebral_shunt

> A cerebral shunt is a device permanently implanted inside the head and body to drain excess fluid away from the brain.

Some of the most annoying moments in my career were occasional coworkers who talked a lot without making a point. They just invoked topics loosely related to the problem at hand, like stories from previous jobs or what they've read on the internet about this problem, until the manager said something like "are you trying to say that..." and then formulated what they thought the point was, except that the speaker clearly never intended to say anything like that. All the other engineers in the room are giving each other side eye but the manager thinks they've just got "a valuable input".

The tactic is simply to talk relevantly until the listener fools themselves into thinking they understand you, except that "understanding" will be an illusion and 100% manager's own analytical effort. And it's honestly embarrassing how often and well that works. Quite a few of those guys were promoted first, too.

I'm pretty sure our upcoming future with AI will be like that quote from forest ranger at Yosemite National Park, on why it is hard to design the perfect garbage bin to keep bears from breaking into it: "There is a considerable overlap between the intelligence of the smartest bears and the dumbest tourists."

The whole idea of a criteria to separate humans from AI is a misconception. I think any human who confuses smooth talking with making sense will be ruthlessly exploited in relatively near future. There are humans who already do this successfully to other humans, what do you think will happen when they get to automate their efforts.

> I think any human who confuses smooth talking with making sense will be ruthlessly exploited in relatively near future

Ads already ruthlessly exploit these people, taking their money in exchange for garbage products. LLM might make this worse, but we have already automated displaying targeted manipulative messages via ad networks. People who can handle manipulative messages from ads should be mostly resistant to manipulative messaging from bots.

> People who can handle manipulative messages from ads should be mostly resistant to manipulative messaging from bots.

Not necessarily. LLMs at the model of GPT-4, and eventually above, affect the economics of advertising. Currently, there's only so much sense in optimizing your ads before it's cheaper to just spam more of it (er, "increase exposure"). LLMs will make it cheaper to optimize the ads further (i.e. get more bang for the same buck), so I expect the baseline manipulative effect of most ads to increase.

Why work on a broad message (ad) when you can afford (cpu time wise) to have a dedicated con artist target every individual in the world at the same time?

Exactly, and thanks for making this point explicitly.

It's one of the end-game consequences of the change in advertising economics I mentioned: with costs of future AI models going down, eventually companies may be able to afford to generate individually personalized and highly persuasive ads on the fly. A personal dedicated con artist for everyone, indeed.

What's more these people often don't know it's even happening.

In fact, the odds are extremely high that you, me, and everyone else here has been taken in by some kind of ad or ad-like rhetoric and internalized it as just being normal world-view.

If there exist those who haven't even been fooled in such a way, they must certainly have the self-awareness to nevertheless entertain doubts on the matter.

Yes, everyone has to one degree or another. No one is an expert in all fields, no one has perfect knowledge and so you will often be presented with things that seem perfectly reasonable but which are entirely false. Worse yet, most of the information you receive will be partly true and partly false with no easy way to tell which is which without engaging in a significant amount of legwork. The trope is the Gell-Mann Amnesia effect which you face every morning when you read a few interesting articles or listen to the daily news. The practices of active deception for sales and influence are all quite clearly laid out by Edward Bernays in his book Propaganda, even our modern concept of "influencers" was known during his day, the more exoteric framing being that people will trust and follow the advice of people that they like or think are similar to them in some way.

I have a new law: "Sufficiently large quantitative change always turns into qualitative change."

> I think any human who confuses smooth talking with making sense will be ruthlessly exploited in relatively near future

Versus the last thousand years?

Just wait until you learn about organized religion it’ll blow your mind.

> Just wait until you learn about organized religion it’ll blow your mind.

That gets super complicated, because arguably religion is less about its content, and more about the community. The hunger for connection isn't going away.

In a sense, religion is the perfect intersubjective phenomenon - the actual beliefs don't objectively matter and can be arbitrary[0], what's important is that everyone in your community subscribes to the same arbitrary set of beliefs. Traditions and proclamations are effective proxies for community cohesion.

(Arguably, national propaganda is the same story, just played on fast-forward.)

Point being, religion is much more than being smooth talked into believing nonsense. It digs roots deep into how humans form societies.

--

[0] - Mostly. Some beliefs are more sticky or self-reinforcing in the real world than others. Religious beliefs undergo natural selection too.

Yes it generalizes well beyond religion. All you need is something that distinguishes you and your group from others, something that is relatively easy to achieve but requires a sign of commitment. It can be how you speak, what you say, what you wear, what you eat, what you say you believe, anything really.

The people I'm less attracted to connect with are religious people, I actually have to force myselft to overlook the religiosity of my mother to live in harmony with her, but it does require effort I rather not do with anybody else. So maybe that's not the whole story, or maybe it's just a modern pitfall that wasn't as important in older times.

> Religious beliefs undergo natural selection too.

All memes do - that's why their name is based on genes. What's interesting is discovering clusters of memes that are self-reinforcing and are passed down consistently, without deviating too much from the original (while still allowing changes), over a long period of time. Arguably, religions built around such clusters have an easier time getting adherents and can get resurrected tens or hundreds of years after being nominally defeated or purged. So I don't think the "actual beliefs (mostly) don't objectively matter" characterization is warranted. I think they do matter a lot from the evolutionary standpoint; not so much for any of the current adherents, but for their grandchildren and later descendants.

To what extent the memetic content carries the institution versus how much an established institution contributes to propagating the content is unclear to me. There's definitely a relationship there, a complex interplay between vested interests, provided services, and people's needs and expectations. As you said, religion is much more than just a set of "nonsense" people get talked into. This makes it hard to discern the real utility of a particular nonsense for ensuring the religion survival.

One rarely discussed peculiarity about religion is that it distorts reality for both believers and non-believers (as one can see by observing how anti-religious people discuss what religion "is", what religious people believe, etc).

Possibly relevant:

LLMs confabulate not hallucinate

https://www.beren.io/2023-03-19-LLMs-confabulate-not-halluci...

Are you a bot? You just submitted the article header back at this person

“Organized religion” describes a lot of things, only a subset of which is actually organized religion.

There are plenty of examples, yes. Abstract art: a bunch of meaningless color patches in which certain viewers manage to find something "deep". Other examples are audiophiles and sommeliers. All those are well known to pay solid sums of money for something they are just imagining, to sellers who managed to exploit this effect.

But creating a new cult (let's call them all "cults") is rare. It takes a person with a unique combination of skills and talents, I bet most of us can't start a religion or succeed at selling snake oil at will. But with help of LLMs...

People who don’t know how to make sense are more likely to be exploited, than to automate their accidental exploits.

> I think any human who confuses smooth talking with making sense will be ruthlessly exploited in relatively near future

Buddy, they already exist, and they're carrying MBAs, Teaching Degrees, and working the Stock Market. Anecdotally, one of the few bona-fide psychopaths I met face to face submitted a thesis that everyone in the school system bent over backwards to say was amazing, on actual review had submitted a complete word-salad.

There's Lots of evidence indicating language models roughly know when they hallucinate.

GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r)

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - [https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305...

Teaching Models to Express Their Uncertainty in Words - [https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205...

Language Models (Mostly) Know What They Know - [https://arxiv.org/abs/2207.05221](https://arxiv.org/abs/2207...

It seems very much like hallucinations aren't a randomness or representation issue. The computation already knows. It just doesn't care about telling you this

I've been using it as a joke but if they're predicting what we want to read, is it possible they just learned to make stuff up by having internet forums in their training data?

The amount of BS we feed each other online, maybe it just thinks we'd prefer something made up than "I don't know"

Not sure if it came off as me saying the parent comment was BS or if what I said was just dumb. Downvotes don't explain anything, haha

For the record I wasn't saying the parent comment was BS :)

RLHF also drops GPT3.5’s chess elo from ~1800 to ~800 which i confabulate to be connected

Is this gpt3.5 Elo a modern chess.com elo ? I feel like there has been some deflation in elo in the last 2 years.

> It just doesn't care

Matrix multiplications cannot feel anything so of course it doesn't care. Humans care. Language models do not.

Also 2 == 2. You're not making an argument, you're just saying that LLMs aren't human.

I knew someone who was like Denys after a stroke. They spoke well, in grammatically correct sentences that made sense even if the content was not rooted in reality. There was not much coherence between sentences or in longer statements but it was more like quick gradual shifts not hard jumps from topic to topic.

So not unlike early LLM chat in areas where the model had not a lot of data about.

Here's the full passage in The Language Instinct about Denyse:

    Here is another interview, this one between a fourteen-year-old girl called Denyse and the late psycholinguist Richard Cromer; the interview was transcribed and analyzed by Cromer's colleague Sigrid Lipka.

    > I like opening cards. I had a pile of post this morning and not one of them was a Christmas card. A bank statement I got this morning!

    < [A bank statement? I hope it was good news.]

    > No it wasn't good news.

    < [Sounds like mine.]

    > I hate . . . , My mum works over at the, over on the ward and she said "not another bank statement." I said "it's the second one in two days." And she said "Do you want me to go to the bank for you at lunchtime?" and I went "No, I'll go this time and explain it myself." I tell you what, my bank are awful. They've lost my bank book, you see, and I can't find it anywhere. I belong to the TSB Bank and I'm thinking of changing my bank 'cause they're so awful. They keep, they keep losing . . . [someone comes in to bring some tea] Oh, isn't that nice.

    < [Uhm. Very good.]

    > They've got the habit of doing that. They lose, they've lost my bank book twice, in a month, and I think I'll scream. My mum went yesterday to the bank for me. She said "They've lost your bank book again." I went "Can I scream?" and I went, she went "Yes, go on." So I hollered. But it is annoying when they do things like that. TSB, Trustees aren't. . . uh the best ones to be with actually. They're hopeless.

    I have seen Denyse on videotape, and she comes across as a loquacious, sophisticated conversationalist—all the more so, to American ears, because of her refined British accent. (My bank are awful, by the way, is grammatical in British, though not American, English.) It comes as a surprise to learn that the events she relates so earnestly are figments of her imagination. Denyse has no bank account, so she could not have received any statements in the mail, nor could her bank have lost her bankbook. Though she would talk about a joint bank account she shared with her boyfriend, she had no boyfriend, and obviously had only the most tenuous grasp of the concept "joint bank account" because she complained about the boyfriend taking money out of her side of the account. In other conversations Denyse would engage her listeners with lively tales about the wedding of her sister, her holiday in Scotland with a boy named Danny, and a happy airport reunion with a long-estranged father. But Denyse's sister is unmarried, Denyse has never been to Scotland, she does not know anyone named Danny, and her father has never been away for any length of time. In fact, Denyse is severely retarded. She never learned to read or write and cannot handle money or any of the other demands of everyday functioning.

    Denyse was born with spina bifida ("split spine"), a malformation of the vertebrae that leaves the spinal cord unprotected. Spina bifida often results in hydrocephalus, an increase in pressure in the cerebrospinal fluid filling the ventricles (large cavities) of the brain, distending the brain from within. For reasons no one understands, hydrocephalic children occasionally end up like Denyse, significantly retarded but with unimpaired—indeed, overdeveloped—language skills. (Perhaps the ballooning ventricles crush much of the brain tissue necessary for everyday intelligence but leave intact some other portions that can develop language circuitry.) The various technical terms for the condition include "cocktail party conversation," "chatterbox syndrome," and "blathering."

Source: http://f.javier.io/rep/books/The-Language-Instinct-How-the-M...

After digging in a little further, it appears the primary source for the anecdote is this book, in a chapter called "The cognition hypothesis of language acquisition". Book:

Cromer, R.F. 1991. "Language and thought in normal and handicapped children." Cambridge, MA: Blackwell

https://www.amazon.com/Language-Handicapped-Children-Cogniti...

> I hate . . . , My mum works over at the, over on the ward and she said "not another bank statement." I said "it's the second one in two days." And she said "Do you want me to go to the bank for you at lunchtime?" and I went "No, I'll go this time and explain it myself." I tell you what, my bank are awful. They've lost my bank book, you see, and I can't find it anywhere. I belong to the TSB Bank and I'm thinking of changing my bank 'cause they're so awful. They keep, they keep losing . . . [someone comes in to bring some tea] Oh, isn't that nice.

In all seriousness, this feels weirdly similar to listening to Donald Trump.

Most transcripts of normal people end up sounding like that. Trump just tweets exactly how he talks, while most people code-switch in text.

I don't know, I've had my nose in (machine-generated) transcripts a lot over the last few weeks, and they didn't give me that feeling, even when they were quite rambly.

I think it's the "jumping from one subject to the next without a broader point" thing that made me think of Donald Trump.

Sorry for the tangent, but the following is surprising to me:

> My bank are awful, by the way, is grammatical in British, though not American, English.

How would someone phrase this in idiomatic American English?

“My bank is awful.”

American English views organizations as a singular entity, not so much a group of individuals.

As does modern Australian English for the most part (I would typically say "my bank is awful". But I'd just as likely use "they" as a pronoun for it, especially if describing actions that were clearly carried out by those working there). I'd guess the same of NZ and Canadian English?

Yep - they while describing their actions. In that respect, Canadian English is practically the same as American English. At least in the NY/Ontario region.

It reminds me of my kids occasionally telling a nonsense story as if it's something real that happened, but where it feels like they're subconsciously just practicing grammar or vocabulary and they're perfectly happy if I just correct their sentences rather than respond to the story.