One reason LLMs "hallucinate" (lie/BS) is that there are subtle biases introduced by the training process. In particular human raters don't know what the model knows, so if the model makes up an answer that happens to be correct then that behaviour is rewarded. Also the internet doesn't have many examples of people saying they don't know, because people who don't know just don't reply.
Also RLHF screws up the logprobs.
How do you mean it screws up the logprobs?
A well pre-trained LLM makes every word a good guess of what the authors of its training set would say, following the words that preceded... in a different context, possibly a few years ago, possibly as part of a web-fiction. "accurate" is the wrong word here.
To make an LLM prefer, when answering factual questions, to rely on factual sources and approximate the real world, you have to use an additional process that does so. RLHF and CAI might be part of the solution to that, or not, depending on the feedback they provide.
A first draft could answer the prompt in any number of ways, then be re-written with another parameter (ethics) in mind. That’s editing, which these LLMs seem suited for, understandably.
Factually, its first draft made a best guess (prediction) of what’s accurate. It has no reason to later “suspect” that a fact can be “more accurate”. More specific, maybe, or scoped to a type of source.