It can reason. To an extent.
Wrong statements about Python are simply less probable than wrong statements about Rust, since there is more Python than Rust in the training data.
That changes exactly nothing about the fact that the system isn't able to detect when it makes a blunder in Python.
That is not what we've observed though. Quite the opposite - we're seeing that the bigger LLM is and the more domain-specific material it digested, the more truthful it becomes.
Yes it can still make an error and be unable to spot it, but so can I.
No, that is not my claim. That is part of the explanation for it.
My claim is this: An LLM is incapable of knowing when it produces false information, as it simply doesn't have a concept of "truthfulness". It deals in probabilities, not alignment with objective reality.
And it doesn't matter how big you make them...this fact cannot change, as it is rooted in the basic MO of language models.
So, now that we have covered what my claim actually is...
> That is not what we've observed though. Quite the opposite - we're seeing that the bigger LLM is and the more domain-specific material it digested, the more truthful it becomes.
...I can ask what this observation has to do with it, and the answer is: Nothing at all. LMs with more params may produce untruthful statements less often, but what does this change about their ability to recignize when they do produce them? And the answer is: Nothing. They still can't.
a LLM can indeed know when it produces likely incorrect responses. Not a hypothetical.
What's the point of making claims you have no intention of rescinding regardless of evidence ? People are so funny.
I claim that the human brain doesn't understand "truthfulness" either. It merely creates the impression that understanding is taking place, by adapting to social and environmental pressures. The brain has no "concepts" at all, it just generates output based on its input, its internal wiring, and a variety of essentially random factors, quite analogous to how LLMs operate.
Do you have any evidence that contradicts that claim?
Empirical evidence? Yes I do.
The brain commands an entity that has to exist and function in the context of objective reality. Being unable to verify it's internal state against that, would have been negatively selected some time ago, because stating: "I'm sure that rumbling cave bear with those big sharp teeth is a peaceful herbivore" won't change the objective reality that the caveman is about to become dinner.
How that works in detail is, to the best of my knowledge, still the subject of research in the realm of neurobiology.
The concept of truth is notoriously hard for humans to grapple with. How do we know something is true isn’t just a neurobiological question, it’s been grappled with throughout the history of philosophy — including major revisions of our understanding in the past 80 years.
And for the record, rumbling cave bears are mostly peaceful herbivores.
For the record, all members of the Genus Ursus belong to the Order Carnivora, which literally translates to "Meat Eaters". And that includes Ursus spelaeus, aka. the Cave Bear.
And while it most likely, like many modern bears, was an Omnivore, that "Omni" very much included small, hairless monkey-esque creatures with no natural defenses other than ridiculously small teeth and pathetic excuses for claws, if they happened to stumble into their cave.
> The concept of truth is notoriously hard for humans to grapple with.
I am not talking about the philosophical questions of what truth is as a concept, nor am I talking about the many capabilities of humans to purposefully reshape others perceptions of truth for their own ends.
I am talking about truth as the observable state of the objective reality, aka. the Universe we exist in and interact with. A meter is longer than a centimeter, and boiling water is warmer than frozen water at the same pressure, whether any given philosophy or fabrication agrees with that or not, is irrelevant.
And as I have demonstrated above, humans, and for that matter other species on this planet featuring capable brains like Corvidae or Cetaceans, do in fact have a concept of truth: They are capable of recognizing false or misleading information as being incongruous with objective reality: A raven that sees me putting food into my left hand, will not jump to a patch of ground where I pretend to put food with my right hand.
This is despite the fact that my actions of "hiding the food" with the empty hand are stochastically indistinguishable from an action of actually hiding food from with my left hand.
I don't think those steps are out of the bounds of possibility, really.
The problem is what you mean when you say "consistency".
The LM checks if sequences are stochastically consistent with other sequences in the training data. Within that realm, the sentence: "In the Water Wars of 1999, the Antarctic Coalitions aramada of Hovercraft valiantly faught in the battle of Golehim under Rear Admiral Korakow, against the Trade Unions Fleets." is consistent. Because, while it is total bollocks, it looks stochasticaly like something that could be in a historical text.
So, in it's context, the LM does exactly what you ask for. It produces output that is consistent with the training data.
Truthfulness is a completely different form of consistency: Does the semantic meaning of the data support the statement I just made? of course it doesn't, there isn't an Antarctic Coalition, there were no Water Wars in 1999, and no one ever built an Armada of Hovercraft for any war against a "Trade Union Fleet".
But to know that, one has to understand what the data means semantically. And our current AIs ... well, don't.
Another wrong statement, you're on a roll today.
https://arxiv.org/abs/2305.11169
https://arxiv.org/abs/2306.12672
There's a word we would use to describe your confidently erroneous output were it one of an LLM. Wonder what that might be..
Then we need a new system, because LMs, no matter if they are large or not, cannot do that, for a very simple reason:
A LM doesn't understand "truthfulness". It has no concept of a sequence being true or not, only of a sequence being probable.
And that probability cannot work as a standin for truthfulness, because the LM doesn't produce improbable sequences to begin with...it's output will always be the most (within heat settings) probable sequence. The LM simply has no way of knowing whether the sequence it just predicted is grounded in reality or not.