Isn't part of the problem that an LLM will complete a statement with words that form a kind of statistical midpoint of the things that might come next. But when we're dealing with specific things in the real world, the true way to complete a sentence will not be that kind of statistical function, in fact it only reliably will follow from actually knowing the answer.

So, it's sort of like driving a car based not on a complete map of the area, but a partial map where the missing pieces are filled in from statistical averages of things that appear generally in the world-- but not necessarily in that location. We all may be impressed at how close the predictions come to reality, but the only truly reliable thing is that those predictions, measured with precision, will be wrong. And if you rely on them without verifying them yourself, you might have a serious problem.