We're full of seemingly weird cognitive biases: Roll a roulette wheel in front of people before asking them the percentage of African counties are in the UN, their answers correlate with the number on the wheel.
Most of us judge logical strengths of arguments by how believable the conclusion is; by repetition; by rhyme; and worse, knowledge of cognitive biases doesn't help as we tend to use that knowledge to dismiss conclusions we don't like rather than to test our own.
The people can see a random number that they know is random, and yet be influenced by it when attempting facts.
> LLM output has not discernible pattern to it, you cannot tell at all whether what it's saying is true or not.
LLMs are the pattern. This is a separate axis to "is it true?"
I'd like a little more background on that claim.
As far as I've been able to tell from my understanding of LLMs, everything they create is a hallucination. It's just a case of "text that could plausibly come next based on the patterns of language they were trained on". When an LLM gets stuff correct, that doesn't make it not a hallucination, it's just that enough correct stuff was in the training data that a fair amount of hallucinations will turn out to be correct. Meanwhile, the LLM has no concept of "true" or "false" or "reality" or "fiction".
There's no meta-cognition. It's just "what word probably comes next?" How is that just "an engineering problem [that] can be solved"?