News for hackers. Stuff that matters.
Re:GPTs and Hallucination by HarHarVeryFunny on Tuesday September 10, 2024 @11:53AM (#41502093)
Are we really still having this conversation in 2024 ?! :-(

Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics.

The useful thing of course is that LLMs often do generate "correct" continuations/replies, specifically when that's predicted by the training data, but it's not like they have a choice of not answering or saying "I don't know" in other cases. They are just statistical word generators - sometimes that's useful, and sometimes it's not, but it's just what they are.

Re:GPTs and Hallucination by DougWebb on Tuesday September 10, 2024 @12:03PM (#41502239)
I've never been worried about LLMs. I've always been worried about how people will use LLMs and how they will interpret the output of LLMs. Especially people who don't understand what LLMs are doing.
Re:GPTs and Hallucination by reaperducer on Tuesday September 10, 2024 @12:17PM (#41502430)
I've always been worried about how people will use LLMs and how they will interpret the output of LLMs. Especially people who don't understand what LLMs are doing.

The problem isn't the people. It's the tech companies.

The tech companies are telling people that it's intelligent, and the tech companies are using it to answer people's questions as if they're presenting facts.

People are using it the way they're told.

If you advertise something as a solution, don't be surprised when people use it to solve things.

Re:GPTs and Hallucination by outofpaper on Tuesday September 10, 2024 @12:41PM (#41502755)
Always remember companies are people too... until we make companies powered by AI. Till that day the underlying problem is always with the people.
Re:GPTs and Hallucination by s_dev on Tuesday September 10, 2024 @12:23PM (#41502528)
Why is this concern more important the what people interpret from the media, social media and the dissemination of information in general where lies and fabrications are also commonplace? Like surely people will always fall for nonsense, lies or fabrications and there is nothing that can be done about that.
Re:GPTs and Hallucination by swatcoder on Tuesday September 10, 2024 @12:49PM (#41502864)
As with all these discussions: accountability and consequence.

We can point at a media company, call out its vested interests, scream about its bias, protest in front of its office, sue it for slander and misrepresentation. We can call out individual personalities the same way. We can strive to drive the companies out of business and the personalities out of work, if we deem it necessary, and we can accumulate a paper trail that holds each one to account.

As neither individuals nor corporate entities, algorithms do not yet carry this kind of legal or public accountability even as we some start to hold them up as oracles. In most cases, failures of an algorithm are treated simply as bugs or user mistakes. Nobody is responsible for anything bad and the so the algorithm can persist and its vendor can shrug off their own responsibility by gesturing towards an perpetual development process instead of accepting consequence: "we work to make the algorithm better every day, try again tomorrow!"

Re:GPTs and Hallucination by s_dev on Tuesday September 10, 2024 @01:12PM (#41503100)
>We can point at a media company, call out its vested interests, scream about its bias, protest in front of its office, sue it for slander and misrepresentation.

Right but the previous election had Russian servers spinning up fake news websites that displayed straight up generated news. Again how do you hold them accountable? You can't the only defence against bullshit is independent thinking.

Re:GPTs and Hallucination by evanmoran on Tuesday September 10, 2024 @12:41PM (#41502752)
The difference is that web/social media is branded as an intelligent being you can ask any question of. We all agree the web is _also_ not reliable, but many people will think GPT / Gemini are verifiably accurate when they aren’t.
Re:GPTs and Hallucination by BeetleB on Tuesday September 10, 2024 @02:02PM (#41503707)
> We all agree the web is _also_ not reliable

You need to expand your circle a bit more :-)

Re:GPTs and Hallucination by jrm4 on Tuesday September 10, 2024 @12:57PM (#41502939)
Not much different. Except that there's a dumb thing that humans are doing which is giving weight to the magic brainy computers.
Re:GPTs and Hallucination by 0xdeadbeefbabe on Tuesday September 10, 2024 @01:01PM (#41502984)
All that training data would make even grep look smart. I'm just glad those magical forest creatures made the data in the first place.
Re:GPTs and Hallucination by codr7 on Tuesday September 10, 2024 @02:05PM (#41503746)
I've found that people in general seem to trust computers more than humans, which made sort of sense for a while.

What they don't fully realize is that this is a completely different game; now the computer is just guessing, as opposed to following a deterministic algorithm to the answer.

And this misunderstanding carries the potential for pretty serious consequences, good luck getting that loan once a computer finds some arbitrary pattern and says no.

Re:GPTs and Hallucination by 6510 on Tuesday September 10, 2024 @03:40PM (#41504699)
If only it would tell you "You've criticized the war effort that day in 2004", in stead it will do parallel construction. The end game will be a kind of SEO for human profiles and we will live happily ever after by the best practice guide lines.
Re:GPTs and Hallucination by Spivak on Tuesday September 10, 2024 @12:36PM (#41502694)
Because LLMs strip away all the context surrounding the information it spits out that let you evaluate its trustworthiness. They're incredibly useful tools, I use them constantly when coding but I can do that because I know enough to validate the information and it happens that the cost of validating the output with the docs is shorter than reading them to find the relevant functions.

I wouldn't dare try to use an LLM for a chemistry question because I wouldn't be able to tell if it makes any sense or not. But if you're not a "tech person" and all you see is some company advertising their AIs as magical knowledge engines with disclaimer text that wouldn't pass accessibility tests, why wouldn't you assume they know their stuff? The Perplexity ads are bordering on negligent.

Re:GPTs and Hallucination by __0x01 on Tuesday September 10, 2024 @01:46PM (#41503514)
This is the most prescient point I have read hitherto regarding LLMs. We are fashioning for ourselves gods of wood and stone.
Re:GPTs and Hallucination by BeetleB on Tuesday September 10, 2024 @02:03PM (#41503716)
s/LLM/Wikipedia/g
Re:GPTs and Hallucination by seba_dos1 on Tuesday September 10, 2024 @12:04PM (#41502246)
Asking "why do large language models hallucinate?" is a good question to ask and answer, just like "why do birds sing?" or "why is the sky blue?" are. The problematic part is when you're somehow surprised that the sky is blue, birds sing and LLMs do nothing but hallucinate.
Re:GPTs and Hallucination by lainga on Tuesday September 10, 2024 @03:54PM (#41504822)
Tiger gotta hunt, bird gotta fly, LLM gotta answer "why, why, why?"

Tiger gotta sleep, bird gotta land, LLM gotta produce a token from the context at hand

Re:GPTs and Hallucination by golergka on Tuesday September 10, 2024 @12:20PM (#41502490)
> Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics.

That's a false dichotomy.

Re:GPTs and Hallucination by Feathercrown on Tuesday September 10, 2024 @12:24PM (#41502541)
Their ability to generate facts is a consequence of the word statistics they are truly using. I think it's fair to say the statistics explanation is a more accurate interpretation.
Re:GPTs and Hallucination by golergka on Tuesday September 10, 2024 @12:27PM (#41502573)
More accurate than what? Statistics "explanation" is something which is technically correct. But it also doesn't present the full picture — for example, the fact that LLMs clearly build an internal mental model of the question they're talking about.
Re:GPTs and Hallucination by HarHarVeryFunny on Tuesday September 10, 2024 @02:37PM (#41504084)
They build internal representations of the input only, and to the extent needed, to get the statistics right. This really isn't a "world model" of factual data, but rather a "source model" of what would various sources (training texts) say.

The responses of the model don't represent what it understands per some internal model, because there is no "it", only models of the sources it was trained on, and it'll just as happily generate lies as truths, or smart vs dumb answers (it's all just words) if that is what its source modelling calls for.

What most people mean when they say the model is hallucinating/bullshitting isn't where it has learnt a lie, but rather where is is operating "out of distribution", and is therefore (unknowingly) generating a mashup from multiple only loosely related/matching source contexts.

Re:GPTs and Hallucination by nyrikki on Tuesday September 10, 2024 @12:42PM (#41502766)
I can offer cite that hallucinations are an innate property of LLMs, can you provide one that shows they have 'an internal mental model'?

https://arxiv.org/abs/2401.11817

Re:GPTs and Hallucination by ctoth on Tuesday September 10, 2024 @12:51PM (#41502881)
Re:GPTs and Hallucination by semi-extrinsic on Tuesday September 10, 2024 @01:44PM (#41503490)
I'm not convinced at all. The only thing they are doing is perturbing some of the model weights in intermediate layers, and seeing if the output of the final layer is consistent with the perturbations. It would be a shitty model if that was not the case.

The fancy part in the paper is figuring out how to perturb the intermediate layers in the way you want. But the findings are not impressive.

Note also that the "probe geometry" stuff is so speculative they left it out of the academic paper completely.

In the same way, it has been known since the 90s that if you take the matrices from Finite Element Models and visualize them as graphs, structures appear that kind of resemble the physical appearance of the object being modelled. Here for instance is for a helicopter:

http://yifanhu.net/GALLERY/GRAPHS/GIF_SMALL/Pothen@commanche...

Yet nobody thinks Finite Element Models have an internal mental representation of the world.

Re:GPTs and Hallucination by FrustratedMonky on Tuesday September 10, 2024 @05:59PM (#41506048)
"Yet nobody thinks Finite Element Models have an internal mental representation of the world."

At this point, I'm not sure some wouldn't argue that.

The difference is, put the AI on a loop, with constant feedback, learning. Instead of just a 'pre-trained' model. Make the actual model, live, always learning, so the context window is infinite. This of course would not be for everyone, because it would take all the resources of the training infrastructure to be focused on one person/view. But that gets closer to the human mind, and at that point, we probably couldn't say for sure that the 'perturbations' aren't experiencing something subjective.

Where is the proof that humans have an internal mental representation of the world.

Re:GPTs and Hallucination by nyrikki on Tuesday September 10, 2024 @01:20PM (#41503210)
We are using different meanings for terms.

Building a world model for a perfect information game is different than building an mental model of the external world.

In context learning is a well known property of LLMs, while real world generalization, often described through the common sense problem is not.

To me, 'mental models' are personal, internal representations of external reality, which LLMs currently lack, being limited to the corpus.

Re:GPTs and Hallucination by accountnum on Tuesday September 10, 2024 @01:58PM (#41503658)
No, you're not. Are you genuinely trying to suggest that LLMs, which can:

- Construct arbitrary text that isn't just grammatically but semantically coherent

- Derive intent, subtle intent, from user queries and responses

- Emulate endless different personalities and their reactions to endless stimuli

- Describe in detail the statics and dynamics of the world, including sight, smell, touch and sound

do not have a model of the external world? What do you think a "corpus" means in this context? How is the "corpus" of sensory and evolutionary data that makes you up in any way different?

LLMs are excellent common sense reasoners, and they generalize just fine. Why exactly do you think they get things _subtly_ wrong? Make up API syntax that looks sensible but isn't actually implemented? In order to make these guesses they need to have generalized, they need an understanding of the structure underlying naming, such that they can produce _sensible_ output even if they lack the hard facts.

Re:GPTs and Hallucination by FrustratedMonky on Tuesday September 10, 2024 @06:12PM (#41506147)
You are correct. We are flooded with studies on AI now, so can't find reference.

But just few months ago, saw example of AI, from video, building an internal representation of the world. An internal model of the world. Everyone saying this can't be done, it already is. Maybe can argue it wasn't an LLM, and then I'd say were nitpicking over which technology can do it or not. We already have example of tying them together, symbols and LLM's.

Might be related. https://www.nature.com/articles/d41586-024-00288-1 https://www.technologyreview.com/2019/04/08/103223/two-rival...

Re:GPTs and Hallucination by ctoth on Tuesday September 10, 2024 @01:45PM (#41503508)
I believe they used the game to show how the same underlying technology would build a mental model not because the game was perfect-information, but because it was easy to probe for without a lot of other unrelated concepts getting in the way.

Perhaps this is what you're looking for (similar technique, larger model?) https://www.anthropic.com/news/golden-gate-claude

> In the “mind” of Claude, we found millions of concepts that activate when the model reads relevant text or sees relevant images, which we call “features”.

> One of those was the concept of the Golden Gate Bridge. We found that there’s a specific combination of neurons in Claude’s neural network that activates when it encounters a mention (or a picture) of this most famous San Francisco landmark.

Sounds like a mental model to me! An internal representation of an external concept which exists in the real world.

Re:GPTs and Hallucination by Iulioh on Tuesday September 10, 2024 @01:10PM (#41503089)
Uhm.

I don't really know if using an "experiment" is the right way to test this?

Like, if you want to test such hypothesis you should first prove that these LLMs were created with that specific function in mind, we are not observing a natural phenomena, it is a manmade(manmade...loosly) algorithm .

You don't know the internal architecture of the system and you are testing in a reality narrow situation, this is basically the scientific method applied incorrectly.

Re:GPTs and Hallucination by rollcat on Tuesday September 10, 2024 @12:04PM (#41502248)
I don't even like the word "hallucination", because it's an anthropomorphism, further suggesting/misleading into an impression that LLMs have anything to do with how a human would reason.

Fancy autocomplete. Useful, yes. Powerful, definitely. But it's just another tool, limited in applications the same way a hammer is - this is the part that many humans seem to struggle accepting.

Re:GPTs and Hallucination by beezlebroxxxxxx on Tuesday September 10, 2024 @12:18PM (#41502461)
The use of "hallucinate" is also my gripe.

Philosophy of language, and Wittgenstein in particular, has more specific words for the output insofar as they mean anything to humans: "senseless" or "nonsense".

Re:GPTs and Hallucination by SpicyLemonZest on Tuesday September 10, 2024 @12:27PM (#41502577)
But I don't think those terms are accurate. If I say "John Doe was born in 1987", that's a perfectly sensible sentence, even if I don't actually know when he was born and even if it turns out he was actually born in 1971. A nonsense sentence would be something like "John Doe cat five green explicate" or "h4stga3jkui7rutjdyrst".
Re:GPTs and Hallucination by beezlebroxxxxxx on Tuesday September 10, 2024 @01:16PM (#41503149)
I don't understand your point.

You've just re-presented one difference between sense and nonsense in philosophy of language. We might call that first phrase wrong or an error or inaccurate if that was the case. We don't need to use a word like "hallucinate" for your second phrase --- we can just call it nonsense.

"Hallucinate" is far more inaccurate and confusing. It paints a false picture that melds 2 distinct things, as you've described, which need not happen at the same time: errors and nonsense.

Re:GPTs and Hallucination by Iulioh on Tuesday September 10, 2024 @01:12PM (#41503107)
I think the term forst originated with the dream-like images of google deepmind, these were really similar to human visual hallucinations, i guess the term stuck
Re:GPTs and Hallucination by leptons on Tuesday September 10, 2024 @02:15PM (#41503854)
It's unfortunate, but it is what it is.

When a human hallucinates, that is due to the brain operating out of spec. When an LLM "hallucinates", it's performing exactly as intended, it just has delivered a nonsensical result which is the consequence of the tech behind the LLM.

Re:GPTs and Hallucination by 0xdeadbeefbabe on Tuesday September 10, 2024 @01:06PM (#41503041)
You aren't going to like this, but don't insult the hammers.
Re:GPTs and Hallucination by dloss on Tuesday September 10, 2024 @01:19PM (#41503193)
Instead of "hallucinating" I would have preferred the term "bullshitting" -- in the Harry G. Frankfurt sense of not caring about the truth of one's utterances. But it's too late for that.

https://link.springer.com/article/10.1007/s10676-024-09775-5

Re:GPTs and Hallucination by beezlebroxxxxxx on Tuesday September 10, 2024 @01:56PM (#41503621)
Using "bullshit" would be interesting, but to me would introduce a backdoor anthropomorphism to describe the output. The picture is still too human.

Isn't Frankfurt's concept of bullshit made up of 2 parts: 1) a distinction between lying and telling the truth, AND 2) the absence of caring about either when speaking when normally it's assumed present?

Part 1 seems to apply, but part 2 wouldn't. It doesn't make sense to talk about GPT "caring" about its output beyond anthropomorphism. No one talks about their computer caring about having correct or accurate output and neither is it assumed. People would think your imagining a demon in the box. Really, it's even odd to say "GPT lied" outside of very specific circumstances.

Re:GPTs and Hallucination by HarHarVeryFunny on Tuesday September 10, 2024 @03:17PM (#41504452)
I think "bullshitting" fits better than "hallucinating" - just keep spitting out words rather than admit ignorance, but maybe the best human analogy is freestyle rapping where one has to keep the flow of words coming regardless!

Maybe we just need to coin a new word for it - "LLM-ing" perhaps ?!

Re:GPTs and Hallucination by FrustratedMonky on Tuesday September 10, 2024 @12:00PM (#41502183)
""Are we really still having this conversation in 2024 ?! :-( ""

</Sarcasm> Are we in 2024 and people are still just saying "duh its statistics, nothing to see here".

Re:GPTs and Hallucination by fnordpiglet on Tuesday September 10, 2024 @12:18PM (#41502456)
I am impressed that with multimodal modals in the hands of everyone people still say this. I mean it’s true on the surface but the ability to take a picture of a data sheet and wiring diagram and get it to give me step by step instructions to wire two components together and an example of the UART protocol in cpp that while not directly functional captures all the essential information I need. That’s amazing and it’s not just word prediction, the source is images of documents with diagrams!

Anyone who does this and comes away jaded has lost the ability to dream.

Re:GPTs and Hallucination by imiric on Tuesday September 10, 2024 @01:10PM (#41503087)
It's not just true on the surface—it's true at its core. All those features you mention are based on statistical relationships. This doesn't mean that these systems can't be very useful, but it also doesn't mean that they are _intelligent_, as much as we like to call them that. They have no understanding of their input or output, beyond being able to pattern match and mechanically decide what the next token should be based on their training data.
Re:GPTs and Hallucination by fnordpiglet on Tuesday September 10, 2024 @01:42PM (#41503462)
I never said it wasn’t statistical at any level so I’m not sure what your point is. My point was almost everything we do is an approximation (I mean, all of science and technology and engineering) and very little of it isn’t. Fake it until you make it is pervasive in human endeavors. Over time the errors and issues will be smoothed away until it’s still there but it’s not noticeable enough to be a major problem.

It being statistical doesn’t make it useless. It not being some metaphysical concept of awareness doesn’t make it useless.

But the multimodal model support puts the lie to it being just a stochastic parrot of language. The domains it works in -is- more abstract than syntax and grammar even if it’s grounded in it. That’s part of the entire power of the model. In the end it leads to a predicted token but the path it takes is more subtle and complex than a simple Markov chain of tokens.

Re:GPTs and Hallucination by imiric on Tuesday September 10, 2024 @02:38PM (#41504095)
> I never said it wasn’t statistical at any level so I’m not sure what your point is.

You said it was true on the surface, and I'm pointing out that it's true at its core. Your amazement at how well it works is based on our inability as humans to find the same patterns in the data as these models do. Ascribing some higher level sense of intelligence or understanding to these systems because of this party trick is anthropomorphizing what they actually are. We would all be better served by keeping in mind how they work, instead of being surprised when they do output the wrong pattern.

> Over time the errors and issues will be smoothed away until it’s still there but it’s not noticeable enough to be a major problem.

Why do you think this is guaranteed? We can keep throwing data and compute at these systems, and while we might continue to get better results, the current systems will never intuitively understand that e.g. 9.11 is lower than 9.2[1], unless those specific examples are in their training data. As we reach the limits of the data we can feed them, and we have to generate synthetic data, there's no reason to believe that the current approaches will ever fix these problems at their core.

> It being statistical doesn’t make it useless. It not being some metaphysical concept of awareness doesn’t make it useless.

I never said it was. I agree that this can be useful, but again, as long as we're aware of their limits. It's good keeping this in mind as we're reaching the peak of inflated expectations of the hype cycle.

[1]: https://towardsdatascience.com/9-11-or-9-9-which-one-is-high...

Re:GPTs and Hallucination by bongodongobob on Tuesday September 10, 2024 @01:42PM (#41503459)
Then you need to explain why statistics isn't enough to be intelligent. We are at the point where that isn't an obvious argument anymore and I'll point to the best models as to why you might be wrong.
Re:GPTs and Hallucination by imiric on Tuesday September 10, 2024 @03:00PM (#41504285)
Because being able to find patterns in large amounts of data will never make the system intuitively understand math[1], or history[2], or any other field. It will always depend on the data and biases we feed it, at least with the current approaches.

You might say that this is how humans learn and demonstrate intelligence, but it's not the same thing. We have the ability to advance our understanding of the universe without being explicitly trained on every topic. Until we can build systems that can do this, I wouldn't label them as intelligent.

But, again, this doesn't mean that they can't be useful.

[1]: https://towardsdatascience.com/9-11-or-9-9-which-one-is-high...

[2]: https://www.theguardian.com/technology/2024/mar/08/we-defini...

Re:GPTs and Hallucination by asdasdsddd on Tuesday September 10, 2024 @12:07PM (#41502284)
I dont get it, what's the unique insight in the article, isn't it just stats and bad data?
Re:GPTs and Hallucination by FrustratedMonky on Tuesday September 10, 2024 @12:13PM (#41502363)
It's just being a bit reductive to just toss AI out as un-interesting because it is just based on statistics.

And, think there is question of 'bad data' so study is bad/invalid, versus studying impact of 'bad data' on responses is the result.

Re:GPTs and Hallucination by asdasdsddd on Tuesday September 10, 2024 @12:17PM (#41502431)
I meant, the answer to the question of why does AI hallucinate is the same answer to the question "why does any statistical system ever produce an incorrect result" which I thought was well established.
Re:GPTs and Hallucination by FrustratedMonky on Tuesday September 10, 2024 @05:54PM (#41506012)
How can it be well established when the LLMs are still actively changing and improving.

Don't think we can say, statistical systems can have incorrect results, hence we no longer need to study how statistical system produce incorrect results, because we already know they can have incorrect results.

Re:GPTs and Hallucination by octonion137 on Tuesday September 10, 2024 @12:03PM (#41502229)
As opposed to what?
Re:GPTs and Hallucination by FrustratedMonky on Tuesday September 10, 2024 @12:04PM (#41502243)
added sarcasm note. it was referring to the parent.
Re:GPTs and Hallucination by sega_sai on Tuesday September 10, 2024 @12:24PM (#41502534)
I do not find this view to be useful.

The are cases from my personal experience, where I asked somewhat esoteric practical questions, that likely do not (seem) to have a clear answer in the web and I have got considerable help from ChatGPT. At some point this dichotomy of 'statistical word generator' vs 'true intelligence' should go away as it's just not useful. (I think these discussions always lead to Chinese Room problem; and IMO at some point it does not matter what 'dumb' process is behind, provided it solves a problem or the system behaves like an intelligent agent)

Re:GPTs and Hallucination by lanternfish on Tuesday September 10, 2024 @12:29PM (#41502609)
Except there is a qualitative difference in the class of knowledge that a statistical word generator and an expert system would generate.

Just because a LLM _can_ offer valuable and insightful information, doesn't mean that it doesn't also hallucinate. The most troubling factor here is that often the hallucinated content also looks like valuable and insightful information, but is just incorrect. This is the use. You have to hold that awareness whenever interacting with these systems.

Re:GPTs and Hallucination by outofpaper on Tuesday September 10, 2024 @12:39PM (#41502732)
Yup exatly. They are dream machines. LLMs without other systems can only work in the flow. The fact that in this word flow the larger LLMs can generate navigation instructions for actual mazes and solve random algorithmicly generated problems doesn't mean they are not hallucinating it just means we're getting wonderfully useful hallucinations.
Re:GPTs and Hallucination by refulgentis on Tuesday September 10, 2024 @12:43PM (#41502782)
Hallucination is a term that means "imagined facts", so it's very hard for me to parse this comment into something meaningful beyond "if we say it always generates hallucination, we can say it always generates hallucinations"
Re:GPTs and Hallucination by jrm4 on Tuesday September 10, 2024 @12:55PM (#41502919)
You Google a restaurant that appears to be open. You go there, and you find that the restaurant is no longer there.

Did Google "hallucinate" a restaurant? Because this is no different.

Re:GPTs and Hallucination by refulgentis on Tuesday September 10, 2024 @12:59PM (#41502969)
We can empirically test if hallucination is a good word for communicating this concept, by checking if people describe(d) that as a hallucination (they don't).

This is all IMHO, I'm not trying to be difficult or nitpick, I just don't understand the idea as communicated. As applied to LLMs, it sounds like hallucination == could be wrong, and this Google example seems further away even when steel-manning, ex. we don't say all Google results are hallucinated.

Re:GPTs and Hallucination by skywhopper on Tuesday September 10, 2024 @01:13PM (#41503122)
It doesn’t mean automatically wrong. It’s just bullshitting. It makes up something that fits a pattern. Depending on the question, the pattern may be right more often than not.

If you ask ChatGPT, “hey is the McDonald’s near my house open at 6pm?” It doesn’t know anything about where you are or if there’s a McDonald’s or what its hours are. It will likely hallucinate that sure, it’s open at 6pm. But when it does so is it “right” in a meaningful way?

Re:GPTs and Hallucination by jrm4 on Tuesday September 10, 2024 @01:35PM (#41503371)
Yup. IMHO, I think "bullshitting" is a much better word than hallucinating and/or getting it right!

Much like real life bullshitters, it is inclined to say something truthful-sounding, but doesn't actually have a strong reliability towards truth per se.

Re:GPTs and Hallucination by randomdata on Tuesday September 10, 2024 @01:37PM (#41503393)
Bullshitting implies intent to deceive. As far as we know, an LLM honestly "believes" (as if you need another rabbit hole) what it says. Delusion, perhaps?

Delusion implies a degree of consistency, though. LLMs can be on point one minute and completely off the rails the next even when prompted with the same prompt. Hallucination fits better here as it speaks to the real-time "perception" (there's another one for you).

An LLM is not a brain, though, so no matter which analogy you choose, it will come with some flaws. Regardless, "hallucination" has moved past analogy territory and now has its own LLM-specific usage with reasonably wide acceptance so the analogy angle is now moot anyway.

Re:GPTs and Hallucination by jhbadger on Tuesday September 10, 2024 @05:17PM (#41505685)
>Bullshitting implies intent to deceive

Not in the formal sense. The philosopher Harry Frankfurt famously distinguished bullshitting from lying because a liar knows the truth and is trying to hide it where a bullshitter is simply trying to sound convincing and may or may not be telling the truth (and may not even know themselves if they are)

https://en.wikipedia.org/wiki/On_Bullshit

Re:GPTs and Hallucination by randomdata on Tuesday September 10, 2024 @06:20PM (#41506197)
In the current formal sense. It may be true the formal sense in 1986 was different. Words do evolve in meaning over time, but since we're talking about right now...

You are right that lying and bullshitting are different. A lie is a false statement with intent. Bullshit is nonsense with intent. A false statement and nonsense may share some similarities, but are ultimately different.

Perhaps nonsense is the word we should be applying to LLMs, but often what they say isn't nonsense, even if only by accident, so that doesn't exactly work either. Regardless, it doesn't matter now. As before, "hallucination" has moved beyond analogy and now has its own LLM-specific usage.

Re:GPTs and Hallucination by alex_suzuki on Tuesday September 10, 2024 @01:40PM (#41503435)
You can ask it „Is the McDonalds nearby open? I live in Brumbledon, Ohio.“ It will do a search, and then confidently state that the McDonalds in Brumbledlon, Ohio is in fact open 24/7. I guess it doesn’t matter that no such town exists.
Re:GPTs and Hallucination by refulgentis on Tuesday September 10, 2024 @08:30PM (#41507059)
That's fair. I'm honestly stunned how little work there's been to incorporate search, in a real way, into products. perplexity, ChatGPT and bing to some extent, that's it.

(Screenshot is my yet to be released flutter app)

https://imgur.com/a/HCVEfkQ

Re:GPTs and Hallucination by BeetleB on Tuesday September 10, 2024 @01:55PM (#41503612)
> Except there is a qualitative difference in the class of knowledge that a statistical word generator and an expert system would generate.

There's a lot of difference between the two, and you don't have to treat it as one or another. It's OK to treat it as something in the middle.

> The most troubling factor here is that often the hallucinated content also looks like valuable and insightful information, but is just incorrect. This is the use. You have to hold that awareness whenever interacting with these systems.

Completely agree, sans the word "troubling". It's not troubling. It is what it is. As long as you keep it in mind when you use it, and treat it as an entity that can be completely wrong, and use it where it's OK to be completely wrong (e.g. when the output is easily verifiable), there's nothing "troubling" with that.

Re:GPTs and Hallucination by airstrike on Tuesday September 10, 2024 @12:57PM (#41502950)
That's not what the GP is arguing, though
Re:GPTs and Hallucination by jrm4 on Tuesday September 10, 2024 @12:58PM (#41502959)
You're stuck on a problem. You grab a random comic book from the shelf and something that is written in the comic book sparks your solution in your head.

How intelligent is the comic book? Is it hallucinating or being correct or what?

The answer is none of those things, right? YOU did the thinking. Not the inanimate object; what it did was a happy coincidence.

Re:GPTs and Hallucination by BeetleB on Tuesday September 10, 2024 @01:57PM (#41503647)
Pointless discussion on semantics.

If a random page in a random comic book gives me the answer I seek 30% of the time, it's incredibly useful. Little effort was spent in seeking the answer, and the 70% of the time it is wrong led to little waste in time.

Now if you put a mechanical arm interface in the middle where I give my query to a machine, and it randomly picks the comic book and page, which answers my question 30% of the time - I have no trouble calling it "intelligent".

Contrast it with Google searches that don't give me the answer I seek, but use up an order of magnitude more of my time.

Re:GPTs and Hallucination by dontlikeyoueith on Tuesday September 10, 2024 @02:10PM (#41503793)
This is just admitting you don't care about knowledge or truth.
Re:GPTs and Hallucination by skywhopper on Tuesday September 10, 2024 @01:08PM (#41503060)
Can you provide some examples of things that aren’t on the web but that ChatGPT helped you with? I’ve yet to see an example that’s not in the likely training set (which includes more than just the public Internet).

But to your point, I disagree: the mechanism matters. Just because you haven’t detected the limitations of the mechanism behind ChatGPT doesn’t mean it’s not there.

Re:GPTs and Hallucination by emptiestplace on Tuesday September 10, 2024 @02:15PM (#41503851)
Perhaps I'm confused, but your question seems contradictory and ambiguous. You first imply (I think?) that ChatGPT is limited to web-sourced information, but then acknowledge the training set includes more than just the public web. Can you please clarify what you are asking?
Re:GPTs and Hallucination by BeetleB on Tuesday September 10, 2024 @02:01PM (#41503685)
My guess is he's referring to things that are out there on the web, but not in the top results and not easy to find.
Re:GPTs and Hallucination by 0xdeadbeefbabe on Tuesday September 10, 2024 @01:04PM (#41503019)
It's like you want to protect our right to hallucinate, because ChatGPT can't.
Re:GPTs and Hallucination by HarHarVeryFunny on Tuesday September 10, 2024 @06:27PM (#41506245)
Sure, but the question posed isn't whether LLMs exhibit intelligence (obviously so, minimally in Chinese Room sense), or can they combine sources (sure, no way to stop them), but why do they hallucinate.

Notwithstanding the amazing things they can do, I don't think it helps understanding by viewing LLMs in too abstract of a way as intelligent agents. After all, in reality they are "just" language models, and hopefully in 2024 the nuance of what they needed to learn to be GOOD language models doesn't need to be explicitly stated every time we discuss them.

Looking at them as language models, it's easy to explain why they hallucinate, are poor reasoners, etc, and IMO does nothing to distract from understanding why they also exhibit intelligence when operating "in distribution".

Re:GPTs and Hallucination by not2b on Tuesday September 10, 2024 @12:44PM (#41502793)
"Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ?"

That's exactly the question the paper attempts to answer: why do LLMs ever get it right? The answer is that on topics where there's a lot of data and a general consensus on what the right answer is, the statistical model will find that answer, and otherwise you get junk. That's why they work so well for people trying to write Python or Javascript, for example.

But I already knew this, you might say. Sure, but the authors produced evidence to back it up.

Re:GPTs and Hallucination by danielbln on Tuesday September 10, 2024 @02:40PM (#41504119)
As context sizes grow, it's easier to add lots of information outside of training data via in-context learning, which should offset that issue quite a bit.
Re:GPTs and Hallucination by jrm4 on Tuesday September 10, 2024 @12:56PM (#41502931)
There's really no "right or wrong" per se -- the question that's really being asked is "to what extent does it resonate with a person?"
Re:GPTs and Hallucination by wslh on Tuesday September 10, 2024 @01:04PM (#41503012)
I think we're all having different experiences and conversations with LLMs, and sometimes framing them purely as "statistical word generators" or expecting them to function like AI in the traditional sense of automated problem-solving might not capture the whole picture.

Like many in this community, I use LLMs daily. My main use case now isn’t software development but rather getting assistance in connecting concepts that I don’t know precisely, guided by my intent. In some ways, it's more akin to "out-of-the-box thinking" but with a tool that helps me explore ideas I might not reach on my own or suffer within the economy of search [1][2]. I might know something about X, Y, and Z, but there's a concept W that ties them all together. Without using W, people in that field might not grasp the connections between X, Y, and Z. Apologies for the abstraction here. I suppose it's my epistemological bias showing!

Even if LLMs don't precisely connect the dots, they help my brain connect them faster.

[1] https://en.wikipedia.org/wiki/Search_theory

[2] https://www.di.ens.fr/~lelarge/soc/varian2.pdf

Re:GPTs and Hallucination by snickell on Tuesday September 10, 2024 @01:15PM (#41503132)
Yes, they are auto-completers, but they are auto-completers that are layered AND operate in higher dimensional spaces. This throws all intuitions off, and I think makes it misleading to think of them as "just" auto-completers. That's part of the story, but not the whole of it.

I suspect we are much closer to auto-completers than most of us like to think, but we're also trained+incentivized by culture, education, parenting, socializing, to produce "useful" results.

Maybe part of the problem is in the data set: how much modeling of "how to admit ignorance or uncertainty" are in LLMs training data sets? If you read the internet, all you see is confident replies to other confident replies. Ignorance or non-confidence tends to elicit either a bluff or non-response. If you read technical literature, you see much of the same.

Maybe LLMs are trained on a dataset, and thereby inherit a culture that's accidentally biased toward ignorant confidence. In human conversation, if somebody asks a question and I don't know the answer, I say I don't know. On the internet, I just skip it and leave it for somebody else who thinks they know.

All this is to say: maybe a statistical autocompleter can admit ignorance instead of firing "neural noise" based on barely-there loose associations. Maybe it just needs a stronger pathway toward talking about not knowing when there's not a strong association.

Re:GPTs and Hallucination by 6510 on Tuesday September 10, 2024 @03:13PM (#41504409)
Wasn't it that it by design prefers things expressed with certainty?
Re:GPTs and Hallucination by HarHarVeryFunny on Tuesday September 10, 2024 @07:17PM (#41506621)
One problem is that the LLM's own knowledge, or lack of it, doesn't follow from any individual training sample(s). Even if there were a bunch of "I don't know X" samples in the training set, that ought to be trumped by one authoritative "X is ..." one.

The next problem is that the LLM doesn't know how reliable it's various training sources are (unlike a human who might trust personal experience > textbook > twitter comment), or even which samples come from which source so that it could learn that.

Re:GPTs and Hallucination by chaosist on Thursday September 12, 2024 @08:06AM (#41519856)
I have thought a lot about this and I suspect "I don't know" would be devastating to the model.

The magic is in the fact that the model can't say "I don't know". The model would have to have arbitrary thresholds and a type of domain/context classification in order to set this arbitrary threshold for the conversation in order to say "I don't know". It would create all these unsolvable boundary conditions. AGI in this context then would be minimizing the need for "I don't know" until it no longer applies. Is that possible? I don't know :)

I would defer to Chomsky also on the subject that we are basically nothing like these models when it comes to language and we are not auto-completers.

Re:GPTs and Hallucination by accountnum on Tuesday September 10, 2024 @02:07PM (#41503765)
Because "truthiness" isn't a magic circuit inside the human brain but a complex set of heuristics. You aren't dealing in facts either, buddy, you deal in approximation and heuristics. Please, explain to me which exact algorithm you use to unify the "facts" you have such magic access to? Can you give me an insight in this 10*12 node tree of grounded logic you purported to be using to make your decisions?

They have a choice, and they literally answer just that. Try asking them what you look like.

You, and so many on this site, use statistics as this magic wand - in combination with 'just' and the almighty quotation marks - to construct comments that superficially look like an argument, but don't contain any reasoning or understanding of the domain you keep on writing about.

Re:GPTs and Hallucination by HarHarVeryFunny on Tuesday September 10, 2024 @02:59PM (#41504283)
> Please, explain to me which exact algorithm you use to unify the "facts" you have such magic access to?

The "magic access" to "facts" (reality) is called embodiment and personal experience with the world in which we live.

An LLM has no embodiment, and therefore no access to reality/facts. All it has is a training set of mixed sources, and a training objective of impersonating those sources (whether an encyclopedia or 4chan comment).

Re:GPTs and Hallucination by og_kalu on Tuesday September 10, 2024 @02:53PM (#41504226)
Pretty much all research (and there's a fair few with different methodologies) on this converge on the same conclusion:

LLMs internally know a lot more about the uncertainty and factualness of their predictions than they say. "LLMs are always hallucinating" is a popular stance but wrong all the same. Maybe rather than asking Why models hallucinate, the better question is to ask "Why not?". During pre-training, there's close to zero incentive to push any uncertainty to the forefront (words).

GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r

Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824

The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734

LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-...

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975

Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334

Re:GPTs and Hallucination by HarHarVeryFunny on Tuesday September 10, 2024 @03:04PM (#41504330)
Yes, because just like chess ELO that we discussed the other day, they need to learn this in order to do well on their training objective - impersonating (continuing) their training sources. If they are continuing a lie then they need to have recognized the input as having this "context", and take that into account during prediction.
Re:GPTs and Hallucination by og_kalu on Tuesday September 10, 2024 @03:10PM (#41504381)
Right but then the problem of hallucination has little to do with statistical generation and much more the utter lack of any incentive in pre-training or otherwise to push features the model has already learnt into the words it generates.
Re:GPTs and Hallucination by HarHarVeryFunny on Tuesday September 10, 2024 @03:35PM (#41504653)
Right, more due to the inherent nature of an LLM than due to that nature being a statistical generator, although as such they amount to the same thing.

One way of looking at it is model talking itself into a corner, with no good way to escape/continue, due to not planning ahead...

e.g. Say we ask an LLM "What is the capital of Scotland?", and so it starts off with an answer of the sort it has learnt should follow such a question "The capital of Scotland is ...". Now, at this point in the generation it's a bit late if the answer wasn't actually in the training data, but the model needs to keep on generating, so does the best it can and draws upon other statistics such as capital cities being large and famous, so maybe continues with "Glasgow" (a large famous Scottish city), which unfortunately is incorrect.

Another way of looking at it rather than talking itself into a corner (and having to LLM it's way out of it), is that hallucinations (non-sequiturs) happen when the model is operating out of distribution and has to combine multiple sources such as the expected form of a "What is .." question reply, and a word matching the (Scottish, large, famous) "template".

Re:GPTs and Hallucination by radarsat1 on Tuesday September 10, 2024 @04:35PM (#41505221)
I think this may be the best explanation I've seen on the topic!

But, shouldn't that situation be handled somewhat by backtracking sampling techniques like beam search? But maybe that is not used much in practice due to being more expensive.. don't know.

Re:GPTs and Hallucination by HarHarVeryFunny on Tuesday September 10, 2024 @06:58PM (#41506473)
Thanks!

I'm not sure if beam search would help much since it's just based on combining word probabilities as opposed to overall meaning, but something like "tree of thoughts" presumably would since I doubt these models would rate their own hallucinated outputs too highly!

Re:GPTs and Hallucination by andrewla on Tuesday September 10, 2024 @03:46PM (#41504765)
> Why would a language model do anything other than "hallucinate"

Indeed, why?!?!?! Why do they so often get the correct answer to very direct questions? Saying "it's in the training data" -- I dare you to find anything in the training data that talks about how many "q's" there are in the word "mortuary", and yet it "hallucinates" up answers to this.

> without any care

What does it mean to care? What question could you ask of an LLM that would allow you to assess how much it "cares" about something?

> but it's not like they have a choice of not answering or saying "I don't know" in other cases

Is it your contention that the phrase "I don't know" has never occurred in the training data for an LLM?

There seems to be a dichotomy of reactions to LLMs. There are technical people saying "it's just an autocomplete engine reciting things from its training set" and there are non-technical people saying "it does more than just next token completion, it is trained to use language".

The first group is technically correct but ignores the fact that it can, emergently, do things far outside of an explainable capability, the second group is technically incorrect but correctly perceives that it can use language in novel ways.

Neither group captures the fact that we built extremely complex linear algebra machines that for reasons we do not understand, despite being trained on an incredibly simple task (next-token-prediction) are capable of actually using language in a way that ten years ago we assumed only humans could do.

Re:GPTs and Hallucination by HarHarVeryFunny on Tuesday September 10, 2024 @04:00PM (#41504899)
> Is it your contention that the phrase "I don't know" has never occurred in the training data for an LLM?

No, but when it does occur in the training data it's a reflection of that particular source/speaker not knowing, which isn't the same as the LLM not knowing because it was also trained on millions/billions of additional sources.

For an LLM to learn to say "I don't know" appropriately, it would need to know when it itself doesn't know (and have that change if you told it), and of course it doesn't have that capability.

Re:GPTs and Hallucination by andrewla on Tuesday September 10, 2024 @04:11PM (#41505001)
> of course it doesn't have that capability

Yes, of course not. Of course. There's no way, for example, that I could ask it if it knew what number I was thinking of, and, after I tell it the number and ask the same question, that it could express that it didn't know before and did know after. Absolutely impossible. Out of the question that it would have this capability. Clearly impossible given its architecture. No way that it could possibly do this task. Why, it would require advances in machine learning and quantum computers and understanding a theory of consciousness and perception at a level that we won't have for centuries. Maybe even a completely new design and training procedure to even begin to approach this insurmountable task.

And if it did demonstrate this ability, then clearly your assessment of its capabilities would be completely wrong and you would have to step back and reconsider how much its training data reflects its abilities.

Re:GPTs and Hallucination by HarHarVeryFunny on Tuesday September 10, 2024 @05:58PM (#41506036)
Well, so far there's no model that knows what it doesn't know, and hallucination continues to be a problem.

So maybe you can enlighten us all, AI labs included, with your genius as to how to solve hallucination, and how to do so only via changes to the training set since that is what you suggest.

To make things easy for you, lets assume that every training text has been augmented with source information such that the model could potentially learn which sources are trustworthy on given subjects or not, and therefore assess whether it knows something or not.

So what else are you going to add to the training set that you claim would induce it to learn this self-referential "I know X" knowledge to better achieve it's next word loss ? Why do you think the AI labs have not done what you are suggesting ?

Re:GPTs and Hallucination by andrewla on Wednesday September 11, 2024 @09:39AM (#41511261)
You misunderstand me -- my position is that we are completely ignorant as to how LLMs do what they do, and it is delusional to think that we understand what their capabilities are or how they are limited.

Until we understand why they sometimes don't hallucinate we can't begin to approach the problem of why the do. And above I directly refute the notion that no model "knows what it doesn't know" -- they are manifestly capable of demonstrating ignorance.

As to next steps in this I have no concrete thoughts, and I don't even know whether LLMs are a dead end. What I suspect is that as we apply more computational power to the problem, as we've seen in the past, computational power plus network depth plus training data means that the structure of the models (like the attention mechanism or the tokenization) will eventually be unnecessary and the structures will be self-discovered and more capable for it.

As models get deeper and more expressive I am confident that things like a broader capability to understand the limitations of its own knowledge or to test its knowledge against objective reality will emerge organically. We may be able to make leaps by creating shortcuts for some of these mechanisms, but in the end depth and computation power will trump all.

Re:GPTs and Hallucination by HarHarVeryFunny on Wednesday September 11, 2024 @05:01PM (#41515309)
> my position is that we are completely ignorant as to how LLMs do what they do

That's simply wrong.

Whole teams of very smart people at places such as Anthropic are working on this exact problem (the field of "mechanistic interpretability"), and have made considerable headway, and have published their results.

Re:GPTs and Hallucination by Nition on Tuesday September 10, 2024 @04:45PM (#41505327)
Just a note: Truthiness means a feeling of truthfulness, even if not actually true. I'd argue that LLMs do care about that, but I suspect you meant 'truthfulness'.