Knowledge without a feedback mechanism for verification is prone to veer off course.
Right now the feedback mechanism is based on human reinforcement.
What if we put an LLM in an android’s body and let the machine interact with the world through the same senses that we do?
Ultimately though this is all about the definition of words. Humans find the distinction between "true" and "false" to be important and meaningful; animals don't care. AIs don't care (being incapable of caring), but humans care very much about the propositional veracity of statements that AI makes.
The important thing is that we're on the same page about what's actually happening with these systems and how they work and don't slip into magical thinking or assume that just because a system is facile with language it automatically has a sense of truth or right or wrong.
This is only going to get more complicated as models get more sophisticated.
https://www.youtube.com/watch?v=yhfl7kasjZc
There are a quite a few videos with the same reaction. There does seem to be something going on.
But I think almost none of them would ascribe "knowledge" to a statistical engine that output propositions of random truth values weighted by how many examples of that proposition it's seen in it's training data.
You can view the logprobs in the API and literally watch it choose to say different things based on it's internal dice rolls.