People in particular have evolved complex self protective mechanisms to provide the right answers for their given environment for safety reasons, based on a number of different individual strategies. For example, the overly honest, the self depreciating, the questioner, the prosecutor, the victim, the liar, the absent minded professor, the idiot, etc.

LLMs are not that complex or self-referential.

Personally, my guess is that you'd want to build a model (of some kind!) whose sole job is determining the credibility of given string of tokens (similar to what someone else noted in a sibling comment about high answer volatility based on minor input changes - that does sound like a signal of low credibility), and somehow integrate THAT self-referential feedback into the process.

Notably, even the smartest lawyers (or perhaps, especially the smartest lawyers) will have assistants do research once they've set out a strategy so they are sure THEY aren't bullshitting. Same with professors, professional researchers, engineers, etc.

Because until someone goes and actually reads the case law from a credible source, or checks the primary research, or calculates things, it's possible someone was misremembering or just wrong.

Being right more often is not about never having a wrong thought/idea/statement, it's about double checking when you're thinking you might be bullshitting, and NOT saying the bullshit answer until you've checked. Which is proportionally, very expensive. The really good professionals will generate MANY lines of such inquiry in parallel for folks to track down, and then based on their degree of confidence in each one and the expected context the answer will be used in, will formulate the 'most correct' response, which is proportionally even more expensive.

So at least during the process, there would be a signal that the system was likely 'bullshitting'. Which might help it in at least being able to signal when it's answers are low-confidence. (human equivalent of stuttering, looking down and away, looking ashamed haha!)

Every human gets fooled sometimes in at least some venue though.