News for hackers. Stuff that matters.
Re:Overcoming the Limits of Large Language Models by darby_nine on Wednesday July 17, 2024 @10:42PM (#40992043)
Man it seems like the ship has sailed on "hallucination" but it's such a terrible name for the phenomenon we see. It is a major mistake to imply the issue is with perception rather than structural incompetence. Why not just say "incoherent output"? It's actually descriptive and doesn't require bastardizing a word we already find meaningful to mean something completely different.
Re:Overcoming the Limits of Large Language Models by jwuphysics on Wednesday July 17, 2024 @11:00PM (#40992115)
> Why not just say "incoherent output"? Because the biggest problem with hallucinations is that the output is usually coherent but factually incorrect. I agree that "hallucination" isn't the best word for it... perhaps something like "confabulation" is better.
Re:Overcoming the Limits of Large Language Models by nl on Thursday July 18, 2024 @12:14AM (#40992460)
And we use "hallucination" because in the ancient times when generative AI meant image generation models would "hallucinate" extra fingers etc.

The behavior of text models is similar enough that the wording stuck, and it's not all that bad.

Re:Overcoming the Limits of Large Language Models by darby_nine on Friday July 19, 2024 @08:17AM (#41005615)
That was a misnomer—hallucination refers to perception, not generation. Completely misled an entire generation of people.
Re:Overcoming the Limits of Large Language Models by Raphaellll on Friday July 19, 2024 @11:49AM (#41007681)
"hallucination" was coined in the context of text generating RNNs. Specifically in this blog post by Karpathy in 2015: https://karpathy.github.io/2015/05/21/rnn-effectiveness/
Re:Overcoming the Limits of Large Language Models by linguistbreaker on Wednesday July 17, 2024 @11:44PM (#40992307)
I appreciated a post on here recently that likened AI hallucination to 'bullshitting'. It's coherent, even plausible output without any regard for the truth.
Re:Overcoming the Limits of Large Language Models by wkat4242 on Thursday July 18, 2024 @02:31AM (#40993013)
While I have absolutely no issues with the word "shit" in popular terms, I'd normally like to reserve it for situations where there's actually intended malice like in "enshittification".

Rather than just an imperfect technology as we have here.

Many people object to the term enshittification for foul-mouthing reasons but I think it covers it very well because the principle it covers is itself so very nasty. But that's not at all the case here.

Re:Overcoming the Limits of Large Language Models by pessimizer on Thursday July 18, 2024 @01:52PM (#40998106)
"Bullshitting" isn't a new piece of jargon, it's a common English word of many decades vintage, and is being used in its dictionary sense here.
Re:Overcoming the Limits of Large Language Models by MattPalmer1086 on Thursday July 18, 2024 @07:09AM (#40994472)
More true to say that all output is bullshitting, not just the ones we call hallucinations. Some of it is true, some isn't. The model doesn't know or care.
Re:Overcoming the Limits of Large Language Models by exmadscientist on Thursday July 18, 2024 @01:23AM (#40992726)
"Hallucinations" implies that someone isn't of sound mental state. We can argue forever about what that means for a LLM and whether that's appropriate, but I think it's absolutely the right attitude and approach to be taking toward these things.

They simply do not behave like humans of sound minds, and "hallucinations" conveys that in a way that "confabulations" or even "bullshit" does not. (Though "bullshit" isn't bad either.)

Re:Overcoming the Limits of Large Language Models by devjab on Thursday July 18, 2024 @02:34AM (#40993026)
I disagree with this take because LLMs are, always, hallucinating. When they get things right it’s because they are lucky. Yes, yes, it’s more complicated than that, but the essence of LLMs is that they are very good at being lucky. So good that they will often give you better results than random search engine clicks, but not good enough to be useful for anything important.

I think what calling the times they get things wrong hallucinations is largely an advertising trick. So that they can sort of fit the LLMs into how all IT is sometimes “wonky” and sell their fundamentally flawed technology more easily. I also think it works extremely well.

Re:Overcoming the Limits of Large Language Models by mewpmewp2 on Thursday July 18, 2024 @03:15AM (#40993228)
But the point is, isn't hallucinating about having malformed, altered or out of touch input rather than producing inaccurate output yourself?

It is the memory pathways leading them astray. It could be thought of a memory system that at certain point any longer can't be fully sure if whatever connections they have are from actually being trained or it or created accidentally.

Re:Overcoming the Limits of Large Language Models by Terr_ on Thursday July 18, 2024 @04:50PM (#40999611)
> isn't hallucinating about having malformed, altered or out of touch input rather than producing inaccurate output yourself?

I suppose so, in the sense that someone could simply be lying about pink elephants instead of seeing them. However it's hard to argue that the machine knows the "right" answer and is (intelligently?) deceiving us.

> It is the memory pathways leading them astray.

I don't think it's a "memory" issue as much as a "they don't operate the way we like to think they do" issue.

Suppose a human is asked to describe different paintings on the wall of an art gallery. Sometimes their statements appear valid and you nod along, and sometimes the statements are so wrong that it alarms you, because "this person is hallucinating."

Now consider how the entire situation is flipped by finding out one additional fact... They're actually totally blind.

Is it a lie? Is it a hallucination? Does it matter? Either way you must dramatically re-evaluate what their "good" outputs really mean and whether they can be used.

Re:Overcoming the Limits of Large Language Models by mewpmewp2 on Thursday July 18, 2024 @06:56PM (#41000516)
To me it's more like, imagine that you have read a lot of books throughout your life, but then someone comes in and asks a question from you and you try to answer from memory, but you get beaten when you say something like "I don't know", and you get rewarded if you answer accurately. You do get beaten if you answer inaccurately, but eventually you learn that if you just say something, you might just be accurate and you will not get beaten. So you just always learn to answer to the best of your knowledge, while never saying that you specifically don't know, because it decreases chances of getting beat up. You are not intentionally lying, you are just hoping that whatever you say is accurate to the best you can do according to the neural connections you've built up in your brain.

Like you ask me for a birthdate of some obscure political figure from history? I'm going to try to feel out what period in history the name might feel like to me and just make my best guess based on that, then say some random year and a birthdate. It just has the lowest odds of being beaten. Was I hallucinating? No, I was just trying to not get beaten.

Re:Overcoming the Limits of Large Language Models by Terr_ on Thursday July 18, 2024 @02:58AM (#40993136)
To offer a satirical analogy: "Lastly, I want to reassure investors and members of the press that we take these concerns very seriously: Hindenburg 2 will only contain normal hydrogen gas, and not the rare and unusual explosive kind which can occur in this highly dynamic and growing field."
Re:Overcoming the Limits of Large Language Models by marcosdumay on Thursday July 18, 2024 @02:30PM (#40998412)
Yes (to the edit), an analogy with making planes safer by only using non-flammable fuels is perfect.
Re:Overcoming the Limits of Large Language Models by Terr_ on Thursday July 18, 2024 @04:55PM (#40999657)
I expect most people have already filled in the blanks, but for completeness:

"Lastly, I want to reassure investors and members of the press that we take these concerns very seriously: The Ford Pinto-II will only contain only normal and stable gasoline, and not the rare and unusual burning kind, which is merely a temporary hurdle in this highly dynamic and explos--er--fast growing field."

Re:Overcoming the Limits of Large Language Models by empath75 on Thursday July 18, 2024 @09:12PM (#41001275)
> When they get things right it’s because they are lucky.

This is transparently wrong. It gets so many things right in a response that the few things it gets wrong are tremendously frustrating. I think people underestimate how much correct "knowledge about the world" is expressed in a typical chat gpt response and focus only on the parts that are incorrect.

If it were wrong about _everything_ at rates no better than chance, we wouldn't even be having this conversation because nobody would be using them.

Re:Overcoming the Limits of Large Language Models by kimixa on Thursday July 18, 2024 @02:25AM (#40992985)
I don't really immediately link "Hallucinations" with "Unsound mind" - most people I know have experienced auditory hallucinations - often things like not sure if the doorbell went off, or if someone said their name.

And I couldn't find a single one of my friends who hadn't experienced "phantom vibration syndrome".

Both I'd say are "Hallucinations", without any real negative connotation.

Re:Overcoming the Limits of Large Language Models by linguistbreaker on Thursday July 18, 2024 @10:16AM (#40995775)
How about "dream-reality confusion (DRC)" ?
Re:Overcoming the Limits of Large Language Models by TeaBrain on Thursday July 18, 2024 @02:16PM (#40998303)
There is no dream-reality separation in an LLM, or really any conception of dreams or reality, so I don't think the term makes sense. Hallucination works fine to describe the phenomenon. LLMs work by coalescing textual information. LLM hallucinations occur due to faulty or inappropriate coalescence of information, which is similar to what occurs with actual hallucinations.
Re:Overcoming the Limits of Large Language Models by darby_nine on Friday July 19, 2024 @08:21AM (#41005652)
"Incoherence" seems like a far more natural fit for what you're describing than a human with a sensory delusion or psychosis.
Re:Overcoming the Limits of Large Language Models by marcosdumay on Thursday July 18, 2024 @02:27PM (#40998399)
Bullshit is the most descriptive one.

LLMs don't do it because they are out of their right mind. They do it because every single answer they say is invented caring only about form, and not correctness.

But yeah, that ship has already sailed.

Re:Overcoming the Limits of Large Language Models by mistermann on Thursday July 18, 2024 @03:32PM (#40998958)
"Sound" minds for humans is graded on a curve, and this trick is not acknowledged, or popular.
Re:Overcoming the Limits of Large Language Models by darby_nine on Friday July 19, 2024 @08:20AM (#41005639)
Hallucinations implies that they do behave like a human mind. Why else would you use the word if you were not trying to draw this parallel?
Re:Overcoming the Limits of Large Language Models by krapp on Friday July 19, 2024 @08:28AM (#41005725)
We're stuck with metaphors for human behavior because the way LLMs operate is so alien and counterintuitive, yet similar enough to human behavior, that we haven't yet developed suitable language to describe it. "Hallucination" gets the point across in general terms, at least.
Re:Overcoming the Limits of Large Language Models by wkat4242 on Thursday July 18, 2024 @02:28AM (#40993002)
Hallucination is one single word. Even if it's not perfect it's great as a term. It's easy to remember and people new to the term already have an idea of what it entails. And the term will bend to cover what we take it to mean anyway. Language is flexible. Hallucination in an LLM context doesn't have to be the exact same as in a human context. All it matters is that we're aligned on what we're talking about. It's already achieved this purpose.
Re:Overcoming the Limits of Large Language Models by freilanzer on Thursday July 18, 2024 @04:03AM (#40993491)
Hallucination perfectly describes the phenomenon.
Re:Overcoming the Limits of Large Language Models by dgs_sgd on Thursday July 18, 2024 @02:15PM (#40998290)
I think calling it hallucination is because of our tendency to anthropomorphize things.

Humans hallucinate. Programs have bugs.

Re:Overcoming the Limits of Large Language Models by threeseed on Thursday July 18, 2024 @03:17PM (#40998836)
The point is that this isn't a bug.

It's inherent to how LLMs work and is expected although undesired behaviour.

Re:Overcoming the Limits of Large Language Models by TeaBrain on Thursday July 18, 2024 @02:22PM (#40998358)
The problem with "incoherent output" is that it isn't describing the phenomenon at all. There have been cases where LLM output has been incoherent, but modern LLM hallucinations are usually coherent and well-contructed, just completely fabricated.
Re:Overcoming the Limits of Large Language Models by darby_nine on Friday July 19, 2024 @08:22AM (#41005663)
Do you have an example? Your statement seems trivially contradictory—how do you know it's fabricated without incoherence showing you this? Isn't fabrication the entire point of generative ai?
Re:Overcoming the Limits of Large Language Models by TeaBrain on Friday July 19, 2024 @09:55AM (#41006634)
It seems that your insistence upon using incoherence is based upon a misunderstanding of the word. Coherence does not mean to be factual, but to be logically ordered. Also, the word "fabricated" is generally used colloquially to describe made up information without factual basis.
Re:Overcoming the Limits of Large Language Models by breatheoften on Thursday July 18, 2024 @02:42PM (#40998517)
I think it's a pretty good name for the phenomenon -- maybe the only problem with the term is that what models are doing is 100% hallucination all the time -- it's just that when the hallucinations are useful we don't call them hallucinations -- so maybe that is a problem with the term (not sure if that's what you are getting at).

But there's nothing at all different about what the model is doing between these cases -- the models are hallucinating all the time and have no ability to assess when they are hallucinating "right" or "wrong" or useful/non-useful output in any meaningful way.

Re:Overcoming the Limits of Large Language Models by darby_nine on Friday July 19, 2024 @08:16AM (#41005604)
They aren't hallucinating in any way comparable to humans, which implies a delusion in perception. You're describing the quality of output by using a word used to describe the quality of input.
Re:Overcoming the Limits of Large Language Models by Slow_Hand on Thursday July 18, 2024 @03:28PM (#40998924)
I prefer “confabulate” to describe this phenomena.

: to fill in gaps in memory by fabrication

> In psychology, confabulation is a memory error consisting of the production of fabricated, distorted, or misinterpreted memories about oneself or the world.

It’s more about coming up with a plausible explanation in the absence of a readily-available one.

Re:Overcoming the Limits of Large Language Models by skdotdan on Thursday July 18, 2024 @09:17PM (#41001296)
This is not what hallucination means in the pre-LLM machine learning literature.
Re:Overcoming the Limits of Large Language Models by adrianmonk on Thursday July 18, 2024 @03:33PM (#40998965)
On a literal level, hallucinations are perceptual.

But "hallucination" was already (before LLMs) being used in a figurative sense, i.e. for abstract ideas that are made up out of nothing. The same is also true of other words that were originally visual, like "illusion" and "mirage".

Re:Overcoming the Limits of Large Language Models by skdotdan on Thursday July 18, 2024 @09:16PM (#41001295)
It’s used incorrectly. Hallucination has (or used to have) a very specific meaning in machine learning. All hallucinations are errors but not all errors are hallucinations.
Re:Overcoming the Limits of Large Language Models by lopatin on Thursday July 18, 2024 @10:37PM (#41001639)
Why is "incoherent output" better? When an LLM hallucinates, it coherently and confidently lies to you. I think "hallucination" is the perfect word for this.