I'm not an expert in LLMs though, this is just my understanding of classifiers in general. Maybe with enough data this consideration no longer applies? I'd be interested to know.
Here's an example of someone doing that for 9.9 > 9.11: https://x.com/mengk20/status/1849213929924513905
4 can be an absolute demonic hallucinating machine.
My best guess is that somewhere close to the root of the problem is that language models still don't really distinguish syntagmatic and paradigmatic relationships. The examples in this article are a little bit forced in that respect because the alternatives it shows in the illustrations are all paradigmatic alternatives but roughly equivalent from a syntax perspective.
This might relate to why, within a given GPT model generation, the earlier versions with more parameters tend to be more prone to hallucination than the newer, smaller, more distilled ones. At least for the old non-context-aware language models (the last time I really spent any serious time digging deep into language models), it was definitely the case that models with more parameters would tend to latch onto syntagmatic information so firmly that it could kind of "overwhelm" the fidelity of representation of semantics. Kind of like a special case of overfitting just for language models.
You absolutely could experiment with pushing it into a denial, and I highly encourage you to try it out. The smollm-entropix repo[1] implements the whole thing in a Jupyter notebook, so it's easier to try out ideas.
There are definitely times when entropy can be high but not actually be uncertain (again synonyms are the best), but it seems promising. I want to build a visualizer using the OpenAI endpoints.
When people in this field compare various methods of quantifying model uncertainty, they often perform what is called rejection verification. Basically, you continuously reject data points where uncertainty is high, and see how average quality of the remaining outputs increases. A good uncertainty estimate is highly correlated with output quality, and thus low-uncertainty outputs should have higher average quality.
We use exactly this approach in our recent benchmark of uncertainty estimation approaches for LLMS [1] and have an open-source library under development [2] which allows for such benchmarking. It also can produce uncertainty scores for a given model output, so ppl in industry can integrate it into their applications as well.
Transformers are generative AI, not classifiers. They throw out a lot of statistics in the service of forward progress and completing the generative task. This project is a rudimentary attempt to regenerate those stats