News for hackers. Stuff that matters.

A comment on Internal representations of LLMs encode information about truthfulness. Parent comment

Re:Internal representations of LLMs encode information about... by justinpombrio on Wednesday October 30, 2024 @11:43AM (#41996377)
> If the program was trained on a thousand data points saying that the capital of Connecticut is Moscow, the model would encode this “truthfulness” information about that fact, despite it being false.

This isn't true.

You're conflating whether a model (that hasn't been fine tuned) would complete "the capital of Connecticut is ___" with "Moscow", and whether that model contains a bit labeling that fact as "false". (It's not actually stored as a bit, but you get the idea.)

Some sentences that a model learns could be classified as "trivia", and the model learns this category by sentences like "Who needs to know that octopuses have three hearts, that's just trivia". Other sentences a model learns could be classified as "false", and the model learns this category by sentences like "2 + 2 isn't 5". Whether a sentence is "false" isn't particularly important to the model, any more than whether it's "trivia", but it will learn those categories.

There's a pattern to "false" sentences. For example, even if there's no training data directly saying that "the capital of Connecticut is Moscow" is false, there are a lot of other sentences like "Moscow is in Russia" and "Moscow is really far from CT" and "people in Moscow speak Russian", that all together follow the statistical pattern of "false" sentences, so a model could categorize "Moscow is the capital of Connecticut" as "false" even if it's never directly told so.

[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by RandomLensman on Wednesday October 30, 2024 @12:06PM (#41996733)
That would again be a "statistical" attempt at deciding on it being correct or false - it might or might not succeed depending on the data.
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by justinpombrio on Wednesday October 30, 2024 @12:19PM (#41996911)
That's correct on two fronts. First, I put "false" in quotes everywhere for a reason: I'm talking about the sort of thing that people would say is false, not what's actually false. And second, yes, I'm merely claiming that it's in theory learnable (in contrast to the OP's claim), not that it will necessarily be learned.
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by RandomLensman on Wednesday October 30, 2024 @12:31PM (#41997075)
Am not sure the second part is always true: there might be situations where statistical approaches could be made kind of "infinitely" accurate as far as data is concerned but still represent a complete misunderstanding of the actual situation (aka truth), e.g., layering epicycles on epicycles in a geocentric model of the solar systems.

Some data might support a statistical approach other might not even though it might not contain misrepresentations as such.

[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by dboreham on Wednesday October 30, 2024 @12:59PM (#41997480)
The human feeling you have that what you're doing is not statistical, is false.
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by RandomLensman on Wednesday October 30, 2024 @01:10PM (#41997645)
Based on what research is that universally true? (Other than base physics like statistical mechanics.)
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by whimsicalism on Wednesday October 30, 2024 @01:18PM (#41997760)
Base physics is all we need to know it is true. Souls are unphysical and we've had reason to be pretty confident about that for at least a century.
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by RandomLensman on Wednesday October 30, 2024 @01:22PM (#41997803)
Yes, physics determins how phenomena work and aggregate but that doesn't necessarily support the specific claim (and we also don't know "all of physics").
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by whimsicalism on Wednesday October 30, 2024 @01:25PM (#41997850)
Doesn't support the specific claim that souls don't exist? We know how atoms/waves interact. We have literally no reason to think there is some other soul-based mechanism.

Of course, maybe induction is false and gravity will reverse in the next 3 seconds after writing this comment and God will reveal themself to us. We have no justified reason to think otherwise other than the general principle that things behave the way we observe them to and will continue to do so.

[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by RandomLensman on Wednesday October 30, 2024 @01:36PM (#41997966)
I see no need for a soul - you brought it up, not me.
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by whimsicalism on Wednesday October 30, 2024 @01:45PM (#41998091)
What would it mean that you are 'what you're doing' is not statistical/arising from base interactions - if not that there is some non-physical entity resembling a soul? You're suggesting some sort of non-material component of humanity, yes?

If not then I'm not even sure what the disagreement is.

[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by RandomLensman on Wednesday October 30, 2024 @04:09PM (#41999646)
Base interactions need not strictly create statistical results in the end.
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by whimsicalism on Wednesday October 30, 2024 @01:19PM (#41997772)
Good luck philosophically defending this dividing line between 'statistical' and not
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by slt2021 on Wednesday October 30, 2024 @12:25PM (#41996996)
but the model doesn't operate on token directly, right? all operations are happening in the embedding space, so these tokens get mapped into manifold and one of the dimensions could be representative of fact/trivia ?
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by ottaborra on Wednesday October 30, 2024 @12:30PM (#41997060)
tangent: any reason to assume it gets mapped to a manifold rather than something that is not?
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by youoy on Wednesday October 30, 2024 @01:47PM (#41998107)
I think "manifolds" in AI are not the same as actual smooth manifolds. For starters I would not expect them to have locally the same dimension across the whole dataset.
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by ottaborra on Wednesday October 30, 2024 @04:45PM (#42000066)
Something to chew on for me. But what is a manifold then if not a topological space that is locally the same as R^(some dimension) ?
[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by youoy on Wednesday October 30, 2024 @05:28PM (#42000577)
What I meant is that I can imagine cases where some part of the dataset may look like R2 and then colapse to have a spike that looks like R1, so it is not a standard manifold where all of it has the same dimension.

Appart from that, these "manifolds" have noise, so that is another difference with the standard manifolds.

[ Parent | on HN ]
Re:Internal representations of LLMs encode information about... by Terr_ on Wednesday October 30, 2024 @01:22PM (#41997813)
Another version/interpretation in this "truth" space is whether a model is capturing multi-part correlations with truthy/falsy signals, much like capturing the "sadness" or "sarcasm" of a statement.

In other words, a model might have local contextual indicators, but not be able to recognize global hard logical contradictions.

[ Parent | on HN ]