It looks like creativity has many steps but being able to come with novel, unprompted stuff is important, as long as you are able to discard the bullshit earlier.
"Hallucination" is only a problem if later layers (or additional networks) can't detect and remove it
Yeah I mean sure. Anything is only a problem if it goes undetected. The issue is that if you rely on statistical model, you’ll always have hallucinations, so you can’t filter statistical output with another statistical model if you need real guarantees.
Many products don’t need those guarantees though.
Hallucination describes the same feature you just called "non deterministic sampling", but exclusively the cases that we don't like. It would be really convenient if we could actually draw that line, but we can't. If non-determinism is a core feature, then that feature will be present in every case; including the ones we find desirable, and the ones we find undesirable.
But here’s a case for the other side: sure, most mistakes are just errors, but evolution happens via “mistakes.” Also, LLM’s often deliberately add add randomness at inference time.
That’s a nice slogan, but it’s a gross oversimplification.
In the natural world, you can say that mistakes in DNA replication leads to evolution, but that’s discounting the entire process of natural selection.
Same with creativity. Look at Picasso. His was a technically brilliant realistic painter at 15, but his work later in life evolved to be more abstract and weird. I don’t think that was the result of mistakes, but rather intentionally breaking patterns he learned in his youth.
I don’t know a whole lot about Picasso’s art, but I imagine the way he evaluated his own work played an important role, in being able to see that sometimes creative accidents are interesting.
Exactly. Don't forget that an important factor in the success of GPT3 was RLHF, which is essentially training the model to produce "hallucinations" that are more acceptable on average to human trainers.
This is a poor definition that only applies to language models trained to be truthful. If you trained a language model to lie, and it told the truth, that would also be a hallucination.
Or if a model was trained to never sound confident, and it made confident, but correct, claims.
My definition is more accurate.
> Yes the sampling procedure is nondeterministic but this is unrelated to hallucinations.
It’s not the only factor, but it’s absolutely related. It’s also really easy to explain in a comment.
For example, if you always sampled the lowest ranked token, the model would always hallucinate (by output mostly garbage)
Top-k sampling doesn’t eliminate all errors, unless you’re just always picking the most likely token. At that point the sampling process is deterministic, but we’ve seen model output be poor with that setting for reasons I explain next.
> that hallucination is a deeper problem
Of course, it’s because the training process itself is nondeterministic. We can’t make a perfect model, it’s just not how statistical models work.
The model doesn't sample a probability distribution of individual "facts"[1] it samples a probability distribution of tokens which are generally parts of words, bits of punctuation etc. That we get "facts" out of it which may even be wrong in the first place is an emergent behaviour because of the attention mechanism.
Totally agree that it's a deeper problem and may be intrinsic to the design of the models and the fact that they are trained on a next word prediction task. Karpathy talks about the models as "dreaming text". In that sense it's not surprising that some of it is whacky. Our dreams are too.
[1] By which I mean atomic things that can be right or wrong
Why? I see them as just sampling errors.
Sure a mistake can spark inspiration sometimes, but creativity is much more than mistakes.
> I understand hallucinating as simply being misaligned with the space humans feel appropriate to interpolate between
These language models are next-token predictors. The way the next token is predicted is by sampling a probability space outputted by the model.
That sampling process can be non deterministic.
Hallucinations are when that sampling results in tokens that come together to create a false or otherwise unintended statement.
You can just as well think of everything a model outputs as a hallucination, but we train the model to output a space what we want them to hallucinate is more likely. Otherwise it just outputs meaningless noise.
“Hallucinate” is really an awful word for what it’s trying to describe.