To apply the scientific method and prove correctness, you need to define the subject first. Meaning is ill-defined, just like consciousness, intelligence, and others. LLMs can clearly grasp higher order abstractions and concepts just by reordering the words, which makes many people argue that being a sentence builder is enough to be intelligent, as they also received most of their intelligence from the same source (words written by someone else - social intelligence).

"Honesty" and other forms of self-introspection is just a high level construct which current models aren't trained for, and likely not a fundamental issue with being a stochastic parrot.