The problem isn't the people. It's the tech companies.
The tech companies are telling people that it's intelligent, and the tech companies are using it to answer people's questions as if they're presenting facts.
People are using it the way they're told.
If you advertise something as a solution, don't be surprised when people use it to solve things.
We can point at a media company, call out its vested interests, scream about its bias, protest in front of its office, sue it for slander and misrepresentation. We can call out individual personalities the same way. We can strive to drive the companies out of business and the personalities out of work, if we deem it necessary, and we can accumulate a paper trail that holds each one to account.
As neither individuals nor corporate entities, algorithms do not yet carry this kind of legal or public accountability even as we some start to hold them up as oracles. In most cases, failures of an algorithm are treated simply as bugs or user mistakes. Nobody is responsible for anything bad and the so the algorithm can persist and its vendor can shrug off their own responsibility by gesturing towards an perpetual development process instead of accepting consequence: "we work to make the algorithm better every day, try again tomorrow!"
Right but the previous election had Russian servers spinning up fake news websites that displayed straight up generated news. Again how do you hold them accountable? You can't the only defence against bullshit is independent thinking.
You need to expand your circle a bit more :-)
What they don't fully realize is that this is a completely different game; now the computer is just guessing, as opposed to following a deterministic algorithm to the answer.
And this misunderstanding carries the potential for pretty serious consequences, good luck getting that loan once a computer finds some arbitrary pattern and says no.
I wouldn't dare try to use an LLM for a chemistry question because I wouldn't be able to tell if it makes any sense or not. But if you're not a "tech person" and all you see is some company advertising their AIs as magical knowledge engines with disclaimer text that wouldn't pass accessibility tests, why wouldn't you assume they know their stuff? The Perplexity ads are bordering on negligent.
Tiger gotta sleep, bird gotta land, LLM gotta produce a token from the context at hand
That's a false dichotomy.
The responses of the model don't represent what it understands per some internal model, because there is no "it", only models of the sources it was trained on, and it'll just as happily generate lies as truths, or smart vs dumb answers (it's all just words) if that is what its source modelling calls for.
What most people mean when they say the model is hallucinating/bullshitting isn't where it has learnt a lie, but rather where is is operating "out of distribution", and is therefore (unknowingly) generating a mashup from multiple only loosely related/matching source contexts.
The fancy part in the paper is figuring out how to perturb the intermediate layers in the way you want. But the findings are not impressive.
Note also that the "probe geometry" stuff is so speculative they left it out of the academic paper completely.
In the same way, it has been known since the 90s that if you take the matrices from Finite Element Models and visualize them as graphs, structures appear that kind of resemble the physical appearance of the object being modelled. Here for instance is for a helicopter:
http://yifanhu.net/GALLERY/GRAPHS/GIF_SMALL/Pothen@commanche...
Yet nobody thinks Finite Element Models have an internal mental representation of the world.
At this point, I'm not sure some wouldn't argue that.
The difference is, put the AI on a loop, with constant feedback, learning. Instead of just a 'pre-trained' model. Make the actual model, live, always learning, so the context window is infinite. This of course would not be for everyone, because it would take all the resources of the training infrastructure to be focused on one person/view. But that gets closer to the human mind, and at that point, we probably couldn't say for sure that the 'perturbations' aren't experiencing something subjective.
Where is the proof that humans have an internal mental representation of the world.
Building a world model for a perfect information game is different than building an mental model of the external world.
In context learning is a well known property of LLMs, while real world generalization, often described through the common sense problem is not.
To me, 'mental models' are personal, internal representations of external reality, which LLMs currently lack, being limited to the corpus.
- Construct arbitrary text that isn't just grammatically but semantically coherent
- Derive intent, subtle intent, from user queries and responses
- Emulate endless different personalities and their reactions to endless stimuli
- Describe in detail the statics and dynamics of the world, including sight, smell, touch and sound
do not have a model of the external world? What do you think a "corpus" means in this context? How is the "corpus" of sensory and evolutionary data that makes you up in any way different?
LLMs are excellent common sense reasoners, and they generalize just fine. Why exactly do you think they get things _subtly_ wrong? Make up API syntax that looks sensible but isn't actually implemented? In order to make these guesses they need to have generalized, they need an understanding of the structure underlying naming, such that they can produce _sensible_ output even if they lack the hard facts.
But just few months ago, saw example of AI, from video, building an internal representation of the world. An internal model of the world. Everyone saying this can't be done, it already is. Maybe can argue it wasn't an LLM, and then I'd say were nitpicking over which technology can do it or not. We already have example of tying them together, symbols and LLM's.
Might be related. https://www.nature.com/articles/d41586-024-00288-1 https://www.technologyreview.com/2019/04/08/103223/two-rival...
Perhaps this is what you're looking for (similar technique, larger model?) https://www.anthropic.com/news/golden-gate-claude
> In the “mind” of Claude, we found millions of concepts that activate when the model reads relevant text or sees relevant images, which we call “features”.
> One of those was the concept of the Golden Gate Bridge. We found that there’s a specific combination of neurons in Claude’s neural network that activates when it encounters a mention (or a picture) of this most famous San Francisco landmark.
Sounds like a mental model to me! An internal representation of an external concept which exists in the real world.
I don't really know if using an "experiment" is the right way to test this?
Like, if you want to test such hypothesis you should first prove that these LLMs were created with that specific function in mind, we are not observing a natural phenomena, it is a manmade(manmade...loosly) algorithm .
You don't know the internal architecture of the system and you are testing in a reality narrow situation, this is basically the scientific method applied incorrectly.
Fancy autocomplete. Useful, yes. Powerful, definitely. But it's just another tool, limited in applications the same way a hammer is - this is the part that many humans seem to struggle accepting.
Philosophy of language, and Wittgenstein in particular, has more specific words for the output insofar as they mean anything to humans: "senseless" or "nonsense".
You've just re-presented one difference between sense and nonsense in philosophy of language. We might call that first phrase wrong or an error or inaccurate if that was the case. We don't need to use a word like "hallucinate" for your second phrase --- we can just call it nonsense.
"Hallucinate" is far more inaccurate and confusing. It paints a false picture that melds 2 distinct things, as you've described, which need not happen at the same time: errors and nonsense.
When a human hallucinates, that is due to the brain operating out of spec. When an LLM "hallucinates", it's performing exactly as intended, it just has delivered a nonsensical result which is the consequence of the tech behind the LLM.
https://link.springer.com/article/10.1007/s10676-024-09775-5
Isn't Frankfurt's concept of bullshit made up of 2 parts: 1) a distinction between lying and telling the truth, AND 2) the absence of caring about either when speaking when normally it's assumed present?
Part 1 seems to apply, but part 2 wouldn't. It doesn't make sense to talk about GPT "caring" about its output beyond anthropomorphism. No one talks about their computer caring about having correct or accurate output and neither is it assumed. People would think your imagining a demon in the box. Really, it's even odd to say "GPT lied" outside of very specific circumstances.
Maybe we just need to coin a new word for it - "LLM-ing" perhaps ?!
</Sarcasm> Are we in 2024 and people are still just saying "duh its statistics, nothing to see here".
Anyone who does this and comes away jaded has lost the ability to dream.
It being statistical doesn’t make it useless. It not being some metaphysical concept of awareness doesn’t make it useless.
But the multimodal model support puts the lie to it being just a stochastic parrot of language. The domains it works in -is- more abstract than syntax and grammar even if it’s grounded in it. That’s part of the entire power of the model. In the end it leads to a predicted token but the path it takes is more subtle and complex than a simple Markov chain of tokens.
You said it was true on the surface, and I'm pointing out that it's true at its core. Your amazement at how well it works is based on our inability as humans to find the same patterns in the data as these models do. Ascribing some higher level sense of intelligence or understanding to these systems because of this party trick is anthropomorphizing what they actually are. We would all be better served by keeping in mind how they work, instead of being surprised when they do output the wrong pattern.
> Over time the errors and issues will be smoothed away until it’s still there but it’s not noticeable enough to be a major problem.
Why do you think this is guaranteed? We can keep throwing data and compute at these systems, and while we might continue to get better results, the current systems will never intuitively understand that e.g. 9.11 is lower than 9.2[1], unless those specific examples are in their training data. As we reach the limits of the data we can feed them, and we have to generate synthetic data, there's no reason to believe that the current approaches will ever fix these problems at their core.
> It being statistical doesn’t make it useless. It not being some metaphysical concept of awareness doesn’t make it useless.
I never said it was. I agree that this can be useful, but again, as long as we're aware of their limits. It's good keeping this in mind as we're reaching the peak of inflated expectations of the hype cycle.
[1]: https://towardsdatascience.com/9-11-or-9-9-which-one-is-high...
You might say that this is how humans learn and demonstrate intelligence, but it's not the same thing. We have the ability to advance our understanding of the universe without being explicitly trained on every topic. Until we can build systems that can do this, I wouldn't label them as intelligent.
But, again, this doesn't mean that they can't be useful.
[1]: https://towardsdatascience.com/9-11-or-9-9-which-one-is-high...
[2]: https://www.theguardian.com/technology/2024/mar/08/we-defini...
And, think there is question of 'bad data' so study is bad/invalid, versus studying impact of 'bad data' on responses is the result.
Don't think we can say, statistical systems can have incorrect results, hence we no longer need to study how statistical system produce incorrect results, because we already know they can have incorrect results.
The are cases from my personal experience, where I asked somewhat esoteric practical questions, that likely do not (seem) to have a clear answer in the web and I have got considerable help from ChatGPT. At some point this dichotomy of 'statistical word generator' vs 'true intelligence' should go away as it's just not useful. (I think these discussions always lead to Chinese Room problem; and IMO at some point it does not matter what 'dumb' process is behind, provided it solves a problem or the system behaves like an intelligent agent)
Just because a LLM _can_ offer valuable and insightful information, doesn't mean that it doesn't also hallucinate. The most troubling factor here is that often the hallucinated content also looks like valuable and insightful information, but is just incorrect. This is the use. You have to hold that awareness whenever interacting with these systems.
Did Google "hallucinate" a restaurant? Because this is no different.
This is all IMHO, I'm not trying to be difficult or nitpick, I just don't understand the idea as communicated. As applied to LLMs, it sounds like hallucination == could be wrong, and this Google example seems further away even when steel-manning, ex. we don't say all Google results are hallucinated.
If you ask ChatGPT, “hey is the McDonald’s near my house open at 6pm?” It doesn’t know anything about where you are or if there’s a McDonald’s or what its hours are. It will likely hallucinate that sure, it’s open at 6pm. But when it does so is it “right” in a meaningful way?
Much like real life bullshitters, it is inclined to say something truthful-sounding, but doesn't actually have a strong reliability towards truth per se.
Delusion implies a degree of consistency, though. LLMs can be on point one minute and completely off the rails the next even when prompted with the same prompt. Hallucination fits better here as it speaks to the real-time "perception" (there's another one for you).
An LLM is not a brain, though, so no matter which analogy you choose, it will come with some flaws. Regardless, "hallucination" has moved past analogy territory and now has its own LLM-specific usage with reasonably wide acceptance so the analogy angle is now moot anyway.
Not in the formal sense. The philosopher Harry Frankfurt famously distinguished bullshitting from lying because a liar knows the truth and is trying to hide it where a bullshitter is simply trying to sound convincing and may or may not be telling the truth (and may not even know themselves if they are)
You are right that lying and bullshitting are different. A lie is a false statement with intent. Bullshit is nonsense with intent. A false statement and nonsense may share some similarities, but are ultimately different.
Perhaps nonsense is the word we should be applying to LLMs, but often what they say isn't nonsense, even if only by accident, so that doesn't exactly work either. Regardless, it doesn't matter now. As before, "hallucination" has moved beyond analogy and now has its own LLM-specific usage.
(Screenshot is my yet to be released flutter app)
There's a lot of difference between the two, and you don't have to treat it as one or another. It's OK to treat it as something in the middle.
> The most troubling factor here is that often the hallucinated content also looks like valuable and insightful information, but is just incorrect. This is the use. You have to hold that awareness whenever interacting with these systems.
Completely agree, sans the word "troubling". It's not troubling. It is what it is. As long as you keep it in mind when you use it, and treat it as an entity that can be completely wrong, and use it where it's OK to be completely wrong (e.g. when the output is easily verifiable), there's nothing "troubling" with that.
How intelligent is the comic book? Is it hallucinating or being correct or what?
The answer is none of those things, right? YOU did the thinking. Not the inanimate object; what it did was a happy coincidence.
If a random page in a random comic book gives me the answer I seek 30% of the time, it's incredibly useful. Little effort was spent in seeking the answer, and the 70% of the time it is wrong led to little waste in time.
Now if you put a mechanical arm interface in the middle where I give my query to a machine, and it randomly picks the comic book and page, which answers my question 30% of the time - I have no trouble calling it "intelligent".
Contrast it with Google searches that don't give me the answer I seek, but use up an order of magnitude more of my time.
But to your point, I disagree: the mechanism matters. Just because you haven’t detected the limitations of the mechanism behind ChatGPT doesn’t mean it’s not there.
Notwithstanding the amazing things they can do, I don't think it helps understanding by viewing LLMs in too abstract of a way as intelligent agents. After all, in reality they are "just" language models, and hopefully in 2024 the nuance of what they needed to learn to be GOOD language models doesn't need to be explicitly stated every time we discuss them.
Looking at them as language models, it's easy to explain why they hallucinate, are poor reasoners, etc, and IMO does nothing to distract from understanding why they also exhibit intelligence when operating "in distribution".
That's exactly the question the paper attempts to answer: why do LLMs ever get it right? The answer is that on topics where there's a lot of data and a general consensus on what the right answer is, the statistical model will find that answer, and otherwise you get junk. That's why they work so well for people trying to write Python or Javascript, for example.
But I already knew this, you might say. Sure, but the authors produced evidence to back it up.
Like many in this community, I use LLMs daily. My main use case now isn’t software development but rather getting assistance in connecting concepts that I don’t know precisely, guided by my intent. In some ways, it's more akin to "out-of-the-box thinking" but with a tool that helps me explore ideas I might not reach on my own or suffer within the economy of search [1][2]. I might know something about X, Y, and Z, but there's a concept W that ties them all together. Without using W, people in that field might not grasp the connections between X, Y, and Z. Apologies for the abstraction here. I suppose it's my epistemological bias showing!
Even if LLMs don't precisely connect the dots, they help my brain connect them faster.
I suspect we are much closer to auto-completers than most of us like to think, but we're also trained+incentivized by culture, education, parenting, socializing, to produce "useful" results.
Maybe part of the problem is in the data set: how much modeling of "how to admit ignorance or uncertainty" are in LLMs training data sets? If you read the internet, all you see is confident replies to other confident replies. Ignorance or non-confidence tends to elicit either a bluff or non-response. If you read technical literature, you see much of the same.
Maybe LLMs are trained on a dataset, and thereby inherit a culture that's accidentally biased toward ignorant confidence. In human conversation, if somebody asks a question and I don't know the answer, I say I don't know. On the internet, I just skip it and leave it for somebody else who thinks they know.
All this is to say: maybe a statistical autocompleter can admit ignorance instead of firing "neural noise" based on barely-there loose associations. Maybe it just needs a stronger pathway toward talking about not knowing when there's not a strong association.
The next problem is that the LLM doesn't know how reliable it's various training sources are (unlike a human who might trust personal experience > textbook > twitter comment), or even which samples come from which source so that it could learn that.
The magic is in the fact that the model can't say "I don't know". The model would have to have arbitrary thresholds and a type of domain/context classification in order to set this arbitrary threshold for the conversation in order to say "I don't know". It would create all these unsolvable boundary conditions. AGI in this context then would be minimizing the need for "I don't know" until it no longer applies. Is that possible? I don't know :)
I would defer to Chomsky also on the subject that we are basically nothing like these models when it comes to language and we are not auto-completers.
They have a choice, and they literally answer just that. Try asking them what you look like.
You, and so many on this site, use statistics as this magic wand - in combination with 'just' and the almighty quotation marks - to construct comments that superficially look like an argument, but don't contain any reasoning or understanding of the domain you keep on writing about.
The "magic access" to "facts" (reality) is called embodiment and personal experience with the world in which we live.
An LLM has no embodiment, and therefore no access to reality/facts. All it has is a training set of mixed sources, and a training objective of impersonating those sources (whether an encyclopedia or 4chan comment).
LLMs internally know a lot more about the uncertainty and factualness of their predictions than they say. "LLMs are always hallucinating" is a popular stance but wrong all the same. Maybe rather than asking Why models hallucinate, the better question is to ask "Why not?". During pre-training, there's close to zero incentive to push any uncertainty to the forefront (words).
GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r
Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734
LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-...
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
One way of looking at it is model talking itself into a corner, with no good way to escape/continue, due to not planning ahead...
e.g. Say we ask an LLM "What is the capital of Scotland?", and so it starts off with an answer of the sort it has learnt should follow such a question "The capital of Scotland is ...". Now, at this point in the generation it's a bit late if the answer wasn't actually in the training data, but the model needs to keep on generating, so does the best it can and draws upon other statistics such as capital cities being large and famous, so maybe continues with "Glasgow" (a large famous Scottish city), which unfortunately is incorrect.
Another way of looking at it rather than talking itself into a corner (and having to LLM it's way out of it), is that hallucinations (non-sequiturs) happen when the model is operating out of distribution and has to combine multiple sources such as the expected form of a "What is .." question reply, and a word matching the (Scottish, large, famous) "template".
But, shouldn't that situation be handled somewhat by backtracking sampling techniques like beam search? But maybe that is not used much in practice due to being more expensive.. don't know.
I'm not sure if beam search would help much since it's just based on combining word probabilities as opposed to overall meaning, but something like "tree of thoughts" presumably would since I doubt these models would rate their own hallucinated outputs too highly!
Indeed, why?!?!?! Why do they so often get the correct answer to very direct questions? Saying "it's in the training data" -- I dare you to find anything in the training data that talks about how many "q's" there are in the word "mortuary", and yet it "hallucinates" up answers to this.
> without any care
What does it mean to care? What question could you ask of an LLM that would allow you to assess how much it "cares" about something?
> but it's not like they have a choice of not answering or saying "I don't know" in other cases
Is it your contention that the phrase "I don't know" has never occurred in the training data for an LLM?
There seems to be a dichotomy of reactions to LLMs. There are technical people saying "it's just an autocomplete engine reciting things from its training set" and there are non-technical people saying "it does more than just next token completion, it is trained to use language".
The first group is technically correct but ignores the fact that it can, emergently, do things far outside of an explainable capability, the second group is technically incorrect but correctly perceives that it can use language in novel ways.
Neither group captures the fact that we built extremely complex linear algebra machines that for reasons we do not understand, despite being trained on an incredibly simple task (next-token-prediction) are capable of actually using language in a way that ten years ago we assumed only humans could do.
No, but when it does occur in the training data it's a reflection of that particular source/speaker not knowing, which isn't the same as the LLM not knowing because it was also trained on millions/billions of additional sources.
For an LLM to learn to say "I don't know" appropriately, it would need to know when it itself doesn't know (and have that change if you told it), and of course it doesn't have that capability.
Yes, of course not. Of course. There's no way, for example, that I could ask it if it knew what number I was thinking of, and, after I tell it the number and ask the same question, that it could express that it didn't know before and did know after. Absolutely impossible. Out of the question that it would have this capability. Clearly impossible given its architecture. No way that it could possibly do this task. Why, it would require advances in machine learning and quantum computers and understanding a theory of consciousness and perception at a level that we won't have for centuries. Maybe even a completely new design and training procedure to even begin to approach this insurmountable task.
And if it did demonstrate this ability, then clearly your assessment of its capabilities would be completely wrong and you would have to step back and reconsider how much its training data reflects its abilities.
So maybe you can enlighten us all, AI labs included, with your genius as to how to solve hallucination, and how to do so only via changes to the training set since that is what you suggest.
To make things easy for you, lets assume that every training text has been augmented with source information such that the model could potentially learn which sources are trustworthy on given subjects or not, and therefore assess whether it knows something or not.
So what else are you going to add to the training set that you claim would induce it to learn this self-referential "I know X" knowledge to better achieve it's next word loss ? Why do you think the AI labs have not done what you are suggesting ?
Until we understand why they sometimes don't hallucinate we can't begin to approach the problem of why the do. And above I directly refute the notion that no model "knows what it doesn't know" -- they are manifestly capable of demonstrating ignorance.
As to next steps in this I have no concrete thoughts, and I don't even know whether LLMs are a dead end. What I suspect is that as we apply more computational power to the problem, as we've seen in the past, computational power plus network depth plus training data means that the structure of the models (like the attention mechanism or the tokenization) will eventually be unnecessary and the structures will be self-discovered and more capable for it.
As models get deeper and more expressive I am confident that things like a broader capability to understand the limitations of its own knowledge or to test its knowledge against objective reality will emerge organically. We may be able to make leaps by creating shortcuts for some of these mechanisms, but in the end depth and computation power will trump all.
That's simply wrong.
Whole teams of very smart people at places such as Anthropic are working on this exact problem (the field of "mechanistic interpretability"), and have made considerable headway, and have published their results.
Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics.
The useful thing of course is that LLMs often do generate "correct" continuations/replies, specifically when that's predicted by the training data, but it's not like they have a choice of not answering or saying "I don't know" in other cases. They are just statistical word generators - sometimes that's useful, and sometimes it's not, but it's just what they are.