Re:Decision models like Jev don't beat LLM-as-a-judge or tra...
by andy99 on Monday October 05, 2026 @07:58AM
(#49963691)
Softmax over logits doesn’t give calibrated probabilities either as a rule. I don’t want to comment specifically on Jev but as a rule it’s very hard to get good calibration because it’s somewhat in tension with minimizing training loss for neural networks, e.g. Guo et al (2017) https://arxiv.org/abs/1706.04599
While I know there are ways to improve calibration, I’d personally want to see a lot of evidence the probabilities were actually more meaningful before trusting them. I agree of course that asking an LLM to provide a confidence estimate is meaningless.
LLM’s are not able to give confidence scores, they make them up.
Your AI driven app is making that up. Your product manager and executive team’s demand for confidence in the UI is a totally fictional cosmetic telling them nothing. Your company sold bullshit confidence to your clients.
I’ve done this for many organizations that “formed a new team to work with the CTO on their AI strategy”, and the trappings are the same
You can have an LLM tell you how much of a schema it was able to get information about. And derive a “confidence” or level of compliance from the completeness of the schema
But this is layers upon layers of cruft that a classification model wouldn’t need