This is a poor definition that only applies to language models trained to be truthful. If you trained a language model to lie, and it told the truth, that would also be a hallucination.

Or if a model was trained to never sound confident, and it made confident, but correct, claims.

My definition is more accurate.

> Yes the sampling procedure is nondeterministic but this is unrelated to hallucinations.

It’s not the only factor, but it’s absolutely related. It’s also really easy to explain in a comment.

For example, if you always sampled the lowest ranked token, the model would always hallucinate (by output mostly garbage)

Top-k sampling doesn’t eliminate all errors, unless you’re just always picking the most likely token. At that point the sampling process is deterministic, but we’ve seen model output be poor with that setting for reasons I explain next.

> that hallucination is a deeper problem

Of course, it’s because the training process itself is nondeterministic. We can’t make a perfect model, it’s just not how statistical models work.

The model doesn't sample a probability distribution of individual "facts"[1] it samples a probability distribution of tokens which are generally parts of words, bits of punctuation etc. That we get "facts" out of it which may even be wrong in the first place is an emergent behaviour because of the attention mechanism.

Totally agree that it's a deeper problem and may be intrinsic to the design of the models and the fact that they are trained on a next word prediction task. Karpathy talks about the models as "dreaming text". In that sense it's not surprising that some of it is whacky. Our dreams are too.

[1] By which I mean atomic things that can be right or wrong