LLMs can't self-correct in reasoning tasks, DeepMind study finds

Re:LLMs can't self-correct in reasoning tasks, DeepMind stud... by thesuperbigfrog on Monday October 09, 2023 @03:46PM (#37824585)

>> If an LLM was trained to always say "I don't know", that'd be a useless LLM, so they're trained to NOT say that, even when they don't actually know.

Do LLMs "know" that they don't know?

Real understanding includes acknowledging what you don't know.

Can the predictive text generation of LLMs recognize that its training set did not include data?

[ Parent | on HN ]

Re:LLMs can't self-correct in reasoning tasks, DeepMind stud... by og_kalu on Monday October 09, 2023 @03:48PM (#37824605)

There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this.

GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975

Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334

Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

[ Parent | on HN ]