Even in a small number of paragraphs language models hallucinate. For example Barack Obama, it’s not hard for me to trick it into saying he was born in Chicago. He was not. But there’s a ton of text about Barack Obama in Chicago. Plus, Michelle was born there. When predicting the most likely next token for “Barack Obama was born in ______” there’s a lot of confounding text.

No, there’s a fundamental issue in that statistical language models don’t start with facts to form sentences, they form sentences based on previous tokens and statistical weights. The weights are not auditable or easily updatable.