>> If an LLM was trained to always say "I don't know", that'd be a useless LLM, so they're trained to NOT say that, even when they don't actually know.

Do LLMs "know" that they don't know?

Real understanding includes acknowledging what you don't know.

Can the predictive text generation of LLMs recognize that its training set did not include data?

There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this.

GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975

Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334

Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

Works for me.

https://chat.openai.com/share/683a5a05-50d9-461e-872a-d16b71...