How outdated information hides in LLM token generation probabilities

Re:How outdated information hides in LLM token generation pr... by diggan on Sunday January 12, 2025 @06:40AM (#42672861)

> In principle it could know that something is more commonly cited based on its training data

Could it? Without explicit training for that, how would it be expected to know it has to be able to count occurrences of something?

[ Parent | on HN ]

Re:How outdated information hides in LLM token generation pr... by sd9 on Sunday January 12, 2025 @07:29AM (#42673099)

I think it would be more vibes based - commonly occurring things would be reinforced more in the weights. Rather than it explicitly counting the number of occurrences.

[ Parent | on HN ]

Re:How outdated information hides in LLM token generation pr... by diggan on Sunday January 12, 2025 @09:03AM (#42673594)

So the probabilities would be skewed towards something, but unless the model could somehow count/infer its own weights, I don't see how it could "introspect" to see if something is more common than something else.

[ Parent | on HN ]