Predictions of what? LLMs aren't predicting anything. They are, as the GP says, generating text that appears to be fluent and coherent (as long as we don't inquire into actual semantic relationships with anything else) in the context of the prompt.
"Previous context" here means text. It does not mean "the actual world". Big difference.
You think you experience the "true" world ? You don't. You experience a slice of it that your brain often further fabricates at parts.
To the birds that feel and sense electromagnetic waves intuitively to guide travels, your model of vision and direction is fundamentally incomplete/incorrect. No one gets the experience the real world.
You must be joking. You really think there's not a big difference between a corpus of text downloaded from the Internet and the entire actual world? Or even the sense-data that humans (or birds, for that matter) take in from the world? (And that doesn't even take into account the actions that humans, and birds, take in the world, and then compare the results with what their internal model predicts in order to update the model.)
I don't even know how to respond to something that is so totally off base.
> You think you experience the "true" world ?
I have made no such claim.
> To the birds that feel and sense electromagnetic waves intuitively to guide travels, your model of vision and direction is fundamentally incomplete/incorrect.
Yes, but an LLM isn't a bird any more than it is a human. LLMs don't experience anything.
>Yes, but an LLM isn't a bird any more than it is a human. LLMs don't experience anything.
Who is arguing whether LLMs are humans ? That's such an irrelevant point. The question is whether they are intelligent and understand which they are by any criteria that is actually testable. Nobody cares about building an artificial human.
>LLMs don't experience anything.
You don't know that
What they just don't care about is communicating being out of distribution or making whack predictions. This becomes a problem for humans because they're perfectly fine making things up when the above fail.
But by all accounts, they do learn to distinguish these things. The computation is very much aware when it is going way off base.
GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824