What they just don't care about is communicating being out of distribution or making whack predictions. This becomes a problem for humans because they're perfectly fine making things up when the above fail.
But by all accounts, they do learn to distinguish these things. The computation is very much aware when it is going way off base.
GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
Predictions of what? LLMs aren't predicting anything. They are, as the GP says, generating text that appears to be fluent and coherent (as long as we don't inquire into actual semantic relationships with anything else) in the context of the prompt.
"Previous context" here means text. It does not mean "the actual world". Big difference.
You think you experience the "true" world ? You don't. You experience a slice of it that your brain often further fabricates at parts.
To the birds that feel and sense electromagnetic waves intuitively to guide travels, your model of vision and direction is fundamentally incomplete/incorrect. No one gets the experience the real world.
You must be joking. You really think there's not a big difference between a corpus of text downloaded from the Internet and the entire actual world? Or even the sense-data that humans (or birds, for that matter) take in from the world? (And that doesn't even take into account the actions that humans, and birds, take in the world, and then compare the results with what their internal model predicts in order to update the model.)
I don't even know how to respond to something that is so totally off base.
> You think you experience the "true" world ?
I have made no such claim.
> To the birds that feel and sense electromagnetic waves intuitively to guide travels, your model of vision and direction is fundamentally incomplete/incorrect.
Yes, but an LLM isn't a bird any more than it is a human. LLMs don't experience anything.
>Yes, but an LLM isn't a bird any more than it is a human. LLMs don't experience anything.
Who is arguing whether LLMs are humans ? That's such an irrelevant point. The question is whether they are intelligent and understand which they are by any criteria that is actually testable. Nobody cares about building an artificial human.
>LLMs don't experience anything.
You don't know that
Nope, they summarized it without reading it. Useful enough for those who want to keep a finger to the pulse of AI without a huge time investment.
God save me from having to read every article on HN.
I’m sorry it wasn’t enough of a response to the article for you, but that wasn’t my intention in writing it.
Not true. They are slowly gaining an understanding of reality by reverse engineering the relationships built into human languages. The only reason LLMs are getting better is because they are better modeling the world. At some point the only way to improve token prediction is to gain an understanding of the world.
Perhaps the people building LLMs are doing this, but the LLMs themselves are not. LLMs are just generating text. They aren't "reverse engineering" anything.
> The only reason LLMs are getting better is because they are better modeling the world.
No, they are getting better at generating text that seems fluent and coherent, as long as you don't inquire into any actual semantic relationships with the world. LLMs can't model the world because they don't even have a concept of "the world". All they have is text.
https://danangell.com/blog/posts/gpt-understands/
> The idea is that back when GPT-4 was being trained for it to really consistently get the next word correct, to do that reliably, it had to do more than just bullshit. It had to do more than guess based on patterns. To get the next word right, it had to truly understand the words coming before it.
Imagine if you were training a transformer to predict numbers in a sequence that came from a sine function. Eventually it would get a pretty good test score and you'd find that internally it had built a decent replication of the sine function.
Now do the same with English text. Throw enough data and computation at it and eventually it'll model a rough understanding of gravity and the shapes of objects, as seen in the different answers between GPT-3.5 and GPT-4.
It's more interesting when the thing doing the understanding is a massive system that has both mastered English and other domains and then incidentally modeled some systems English can describe. An analog circuit that models artillery trajectories "understands" one kinematic equation. But that's kind of obvious and mundane. Understanding just one equation barely registers as profound and without that understanding existing in a larger context the word barely seems applicable.
But that confidence is based on intuition, not on any empirical observations. All the research we've seen so far points in the other direction: sufficiently trained LLMs do have a world model that you can find in their weights, changing these weights changes their completions in a way consistent with the new world model, etc. See OthelloGPT for the most blatant example.
I'm starting to wish we had something like a FAQ to point to, with a summary of all the research on the subject, because it's pretty settled by now.
This was always the conjecture of the “ghost in the machine” and mech futurist sci-fi writers.
Would you be willing to provide some reading about this? I’d love to see 1-2 things that could help me (and others) know more about what you know.
I'm not asserting that they "couldn't possibly". I'm asserting that they don't--as in, none of the LLMs we have today have a world model. Perhaps someone might invent a different something in the future that they call an "LLM" that does have a world model, but no such thing exists now.
> that confidence is based on intuition, not on any empirical observations
No, it's not based on either of those. It's based on the explicit descriptions of how existing LLMs work by the people who made them. Their descriptions make clear that LLMs are just confabulating text based on a given prompt and their training data. The LLM doesn't make any connection at all between the text and anything else.
Contrast, for example, with Wolfram Alpha. If you give Wolfram Alpha the prompt "What is the distance from New York to Tokyo", it doesn't confabulate text based on that prompt and some corpus of training data. It first parses the text and infers that the correct response involves a lookup in its geographic database (note that you can skip this step by explicitly selecting the particular tool that does this); then it does the lookup; then it renders the response into text form and outputs it. That is what having a (rudimentary) world model looks like. LLMs do nothing of the sort.
I really can't stand these low-effort takes. It's like saying "Human brains are just meat". You can summarize anything badly, but it's not helpful to the conversation.
Whatever quibbles you have aside, LLMs are generating useful text, today. At some point you're just pointlessly privileging meat over silicon.
No, it isn't, because human brains have rich semantic connections to the rest of the world. LLMs do not. So "LLMs are just generating text" is a justified description of their limitations, in a way that "human brains are just meat" is not.
Prove it
See, for example, my contrast of what an LLM does with what Wolfram Alpha does upthread.
I have no idea why you think Wolfram Alpha is relevant. LLMs can use tools like langchain to do the exact same thing Wolfram Alpha does. What proof do you have for your original claim?
I always find this point a bit odd because humans aren't "optimized" to be correct either.
> "LLMs have no understanding of the underlying reality"
I struggle with this one because I see both sides of it. I was making a prompt the other day and gave a CSV file as an input and told the LLM if could only answer with values from one column and it did exactly as I asked. It's hard for me to see things like that and not believe it has an understanding at some level.
It depends which subsystem.
The older the system in the body the less likely it is to have a high error rate, otherwise we'd die from cancer at a much higher rate or injure ourselves far more often. Of course this also depends on the definition of 'correct', if the system never changed we'd never evolve.
>> "LLMs have no understanding of the underlying reality"
This statement has always bothered me because it really depends on what you mean by 'underlying reality'. How many layers are we talking about? What does understanding mean? Because at the end of the day, humans don't really understand 'underlying reality, we just have a particular set of input devices we take in information and do some transformations on it... Where is the understanding happening?
Edit: there is also loads of evidence that our brains are wired to make incorrect decisions in many cases. Loss aversion is one example.
We recently started providing perception inputs, just like you’re talking about.
Obviously it’s nascent and not exactly a revolutionary statement, but it’s interesting to see the progression towards our experience.
I’m also interested in how the analog computing revolution is going to converge with AI in coming years, as we are making huge inroads into analog computing and that means cheap ubiquitous sensor inputs right in time for AI to start maturing.
Also, given how many words humans have written describing every part of the experience, LLMs can generate a pretty good understanding of it as-is.
I think about this difference a lot, and a fair bit of AI/ML attempts to reproduce this process to some degree.
Humans at least have a concept of "being correct", even if we don't always set that as our primary goal when communicating.
LLMs don't even have a concept of "being correct". That would require having a concept of an "external world" that text refers to, which LLMs don't have. All they have is the text in their training data.
They clearly do as plenty research indicates. You've just decided not to accept this. Pure confirmation bias in action.
I have seen plenty of researchers claim this. What I have not seen is actual support for such a claim.
I have not read much compelling evidence though I have seen a lot of researchers conjecturing and some inappropriately reaching (and anthropomorphising the living hell out of ai in the process sometimes).
There’s a lot to learn and I don’t think anyone can keep up with all the papers coming out so let’s keep things positive and work together to learn as much as we can og_kalu. Thanks for letting me tag this comment on the end of thread.
This is HN. We’re all nerds here anyways and nobody can keep up with everything in a field moving as fast at AI is these days. :)
You're def a good boy!
I would argue, human reasoning is conceptually not so much different from LLMs as one might think.
Wrong. Human beings can act on the world as well as perceive it, and humans have rich internal models of the world that get updated continously as a result of those rich two-way interactions.
> I would argue, human reasoning is conceptually not so much different from LLMs as one might think.
I would argue that this is either an incredible over-estimate of LLMs, or an incredibly impoverished view of humans.
Conceptually this is the same for LLMs.
Input/perceive -> human/LLM -> output/act. You might say the acting part for LLMs is not autonomous but that is only dependent on how much agency is given to LLMs. LLMs outputs are massively used, hence their outputs lead to actions.
> and humans have rich internal models of the world that get updated continously as a result of those rich two-way interactions.
What is different from LLMs: rich, continuous. I would argue these are quantative and qualitative traits that fall outside of what I meant with "conceptually".
No, it isn't. LLMs don't look at the results of their actions and update a world model. Humans do.
One of the things that has been on my mind of late is the idea of how much of what humans call “correct” or “right” is actually socially generated pressures to conform.
Berns and Asch’s various research on social conformity around what we normally would consider objectively correct answers seems to have some potential applications worth testing in AI land. AIs may be much more performant with AI “social inputs” (and of course all the long term memory improvements currently lacking required to use it as a tool). I am very much excited about the network AI and pipeline work (small teams!) currently coming into vogue. Feels like the “right direction” to make incredible progress.
https://www.psychologytoday.com/us/blog/am-i-right/201404/th...
Otherwise agree with your points exactly. Good comment.
And the only reason that is even a thing is that we humans have a concept of actually being right, which we can distinguish from "responding to socially generated pressures to conform".
We seem barely capable of objective thought at best.
There is research that is claimed to show this, but it doesn't. All it actually shows is that the process of making a decision takes time, and involves many different parts of the brain. There is no separate "rationalization" that comes after the decision is made. It's all part of a single process that can't be decomposed into such separate parts.
In any case, all this is irrelevant to what I said. I said we humans have the concept of "actually being right". We don't always live up to that concept, but even making that observation, as you have, shows that we have the concept. If we didn't, we could not even make statements like this one of yours: "our brains care far more about achieving and maintaining status than about being right."
> We seem barely capable of objective thought at best.
I think you have a sadly impoverished view of human thought.
Regarding the research that supports my claim, studies have been done to isolate the different parts of the brain where they tell one part to do something like get water, then they ask the other part of the brain why they got the water and it makes up a reason because it's not aware of what happened in the other part. This is eerily similar to LLMs hallucinating.
This genetic and cultural evolution working together has shaped our brains to care for others, react to those who try to harm us, and to create moral rules that help us to live together successfully.
I’m not sure why responding to socialization pressures to conform to certain behaviors would preclude intrinsic or genetic morality. We know that babies exhibit intrinsic morality from a very young age, that moral behavior exists in non-human species, and that we have parts of the brain just to deal with moral judgement (though we are really in the infancy of understanding and mapping this from what I’ve read).
That’s not what I meant to imply or what research supports.
LLM like have some form of understanding of the world, not sure if its anywhere near comparable to our understanding though. But they are still not general enough to really focus on facts. They can get close but the nature of statistics means there isn't hard checks to truth.
I imagine a lot of progress with AI will involve similar networks, with governance providing evolutionary paths and guidance for multiple concurrent goals.
There are also lots of issues with training data and memory limits that make LLMs weak at continuity. Holes or weaknesses in the training data might “come through” in the behavior of hallucination.
Where do you think general AI is going right now that we should be looking at? I’m overwhelmed by how large this field has become in the last five years and am always interested in what others know.
Why we baby AI on this front, I have no idea.
Hallucinating on the other hand feels way more abstrac. We speak of humans hallucinating answers rarely enough that they can use it for AI with a straight face. Heck, when we do talk about humans hallucinating it's often-as-not in the context of mind expanding experiences; maybe AI hallucinations are even a good thing!
It may be part of complex networks and intelligence, and humans may have a lot of it happening all the time but have ways of compensation or suppression of neurotic or hallucinating thoughts.
We have some intense “mental illnesses” that look a lot like what AI is presenting, where it can’t tell what is going on and fills in the gaps (schizophrenia maybe) or uses behavioral patterns to compensate for memory or intelligence inadequacy.
Obviously straight conjecturing here.
Not if people need to be reminded that, as you say, LLMs are not designed to give reliable answers. Many, many people appear to believe that they are, since they rely on the answers in all kinds of contexts. For example, in past HN threads on LLMs, I have seen people say that they rely on LLM-generated code.
;)
The "verified by me" part is crucial, though, yes?
I'm talking about people, whom I have seen posting here on HN in other threads on this topic, who leave out the "verify" step and just rely on the LLM code as it comes.
To be more specific, I trust GPT-4 to handle tasks like "Rewrite this C++ struct into Rust". I still verify that it's correct as part of a normal code review process, but I trust GPT-4 in that scenario at least as much as I would a junior dev.
LLMs absolutely have concepts that extend beyond just words. As early as 2008 (back when there were no large language models, only "regular" language models), we've been able to demonstrate things that seem to me like the model is learning abstract concepts. For a classic example, see Linguistic Regularities in Continuous Space Word Representations [0], a 2013 paper that talks about a model with a latent space where one can take the vector representations of the words "king", "queen", "man", and "woman" and literally perform the arithmetic `king - (man - woman) ≈ queen`. To me, this clearly demonstrates that the model "understands" the concept of gender, represented by a vector in the model's latent space. The set of numbers that you get from the subtraction `man - woman` represent the concept of the difference between the male and female genders, without needing a specific word tied to that representation of the concept.
It's debatable whether or not that counts as "understanding", but that's more of a semantic debate about what it means to understand, not a debate about the model's internal knowledge of the world and ability to do things with that knowledge.
I don’t mean to discount your excellent comment. I mean obviously these models are so complex already that their black box nature makes it difficult to derive conclusive results, but Arkham’s razor would imply that the simplest explanation — that is it recombined training data creating a heirarchy with multiple factors that provides context for the “correct” answer — that is more likely correct.
In other words a LlM can confer multiple related associations to output based on training data, and that is the most likely explanation for that behavior.
Just look at the poor performance with GPT-3 and the underlying data quality issues the Alpaca dataset team has discussed.
I’m no expert, and don’t mean to counter you as much as contribute my own limited thinking for discussion here.
Cheers!
The essence of bullshit (as he defines it) is that it is neither truth nor lie but rather unconcerned with veracity altogether. A bullshitter wants to appear smart, or convince others to agree with them, or some other conversational objective. Bullshit statements may or may not ultimately turn out to be true, but the thing that makes them bullshit is that the bullshitter didn't actually know when they said it.
Hence I propose a far more succinct term to describe the phenomena at hand. A far more accurate terminology than the presently vogue "hallucinating": AB. Artificial Bullshit.
1) Hallucinations often appear because LLMs are designed to create fluent, coherent text.
2) LLMs have no understanding of the underlying reality that language describes.
3) LLMs use statistics to generate language that is grammatically and semantically correct within the context of the prompt.
It sacrifices accuracy for being good at conversations as it is designed to do. All these criticisms of hallucinations are missing the point.
Generative AI is generative and basically is a specialist at making things up. It’s going to take things like multiprompting, network AIs that fact check output, and a host of other technologies or even entirely new models of AI to solve these problems, but don’t make the mistake of thinking that the system is supposed to be working without hallucinations right now — that’s not what it’s optimized for.