Absolutely correct, and I believe anyone working on these models would agree and, other than as a fun demo, would never suggest that the raw model output gets used for any real purpose. A similar analogy would be self-driving cars. Somewhere "under the hood" there is an ML computer vision model, but it's not like the output layer is just hooked up to the gas and steering. There is all sorts of other logic to make sure the car behaves as intended and fails gracefully under ambiguity.
People see these language models and their flaws and somehow interpret it as a flawed overall product, when they are instead just seeing the underlying model. Admittedly, openAI hasn't helped much by building and promoting a chatbot the way they have.
Lots of cool potential for large language models, very little that comes from raw interaction
It seems like this is only a single layer to something that should be larger. It should be able to tell if what it's saying is true, or to go out and find facts when it's missing them.
The fact that it's only a language model probably means that this is just out of scope.