Even in a small number of paragraphs language models hallucinate. For example Barack Obama, it’s not hard for me to trick it into saying he was born in Chicago. He was not. But there’s a ton of text about Barack Obama in Chicago. Plus, Michelle was born there. When predicting the most likely next token for “Barack Obama was born in ______” there’s a lot of confounding text.

No, there’s a fundamental issue in that statistical language models don’t start with facts to form sentences, they form sentences based on previous tokens and statistical weights. The weights are not auditable or easily updatable.

And any kind of "tuning" of the weights to get better results is going to be so bespoke that you might as well just curate the response yourself

So LLM's are pretty much Mark V Shaney at heart, only with enough weights to get to the far side of Eliza?

Though for that matter a ChatGPT based product would be even more vulnerable to the same thing that made Google search suck: content sources gaming its process