I would agree with you. In general, humans have still not resolved any certain theory of knowledge for ourselves! How can we expect a machine to do that then?

In reality humans are wrong basically most of the time. Especially when you go off a humans immediate reaction to a problem which is what we force LLMs to do (unless you're using chain of thought or pause tokens).

That being said there still is a notion of truthfulness because LLMs can also be made to deceive in which case they 'know' to act deceptively.