Mathematicians build this, and they love the beauty of it so much that they loose touch of reality.
It's not necessarily because the model is wrong. I've had trouble getting useful calibration out of models with an F1 of 0.9. The fundamental problem is twofold. First, it turns out that [0.0, 1.0] is a much larger set than {0, 1}. Second, the kinds of use cases where you care about calibration tend to be fussy and demanding.
As a fun anecdote, I got a chance to ask the person who invented the method scikit-learn uses for calibrated SVMs if he had any advice, and his answer was basically, "good luck."
All that said, sometimes you don't actually need calibration; you just need decent ranking. "Items with a score of 0.9 should be more likely to be positive than ones with a score of 0.5," is an easier requirement than "90% of items with a score of 0.9 should be positive." But I've had trouble getting that out of highly non-linear neural models, too, because they oftentimes produce results where the relationship between score and probability of being in the positive class is not even remotely monotonic.
But I haven't poked at Jev like this either, so I do have to allow that maybe they've found the secret sauce.
LLM’s are not able to give confidence scores, they make them up.
Your AI driven app is making that up. Your product manager and executive team’s demand for confidence in the UI is a totally fictional cosmetic telling them nothing. Your company sold bullshit confidence to your clients.
I’ve done this for many organizations that “formed a new team to work with the CTO on their AI strategy”, and the trappings are the same
You can have an LLM tell you how much of a schema it was able to get information about. And derive a “confidence” or level of compliance from the completeness of the schema
But this is layers upon layers of cruft that a classification model wouldn’t need
While I know there are ways to improve calibration, I’d personally want to see a lot of evidence the probabilities were actually more meaningful before trusting them. I agree of course that asking an LLM to provide a confidence estimate is meaningless.
The benefits of Jev, and the reason I'm excited about them:
* Pre-trained/tuned to provide only structured output
* Small, fast, essentially trivial to locally run - this alone makes them an actual candidate for real-world planning systems. Laya is a local-first 400 million parameter model, for example.
* Their confidence scores are useful, even if those scores are not correct/true probabilities.
For me and my work, they look like nearly turn-key, tiny, fast, locally-hostable models that can manage state transitions in a deeply autonomous system.
LLM-as judge requires, well, what we know as a full LLM. A fully trained classifier model requires fully training - something you cannot / won't do for a embedded autonomous planning system (by contradiction - if this worked, we'd have used it everywhere already!). Totally unfair comparison in my mind.
And how is the confidence score any different than the softmaxed token probabilities you get out of running every llm?
With llama-server, you can use basically any GGUF model as a Jev-like classifier. See Pi.dev's codemode and the accompanying llama backend to the "classify" function. The advantages of Jev-like inference are that:
- You generate 1 token per question in parallel, instead of sequentially generating a bunch of JSON punctuation. So you get much better latency and utilization.
- You can use a custom "sampler" that sees the raw logprobs for these tokens, including all the tokens that might have been generated. So you can generate some "probability" or "confidence" numbers based on how likely the model was to have generated a different answer.
So really, no difference? It's just a nice API with some computational efficiencies.
You're asking me how a structured output which assigns probabilities to classifications of the input text is different than the next-token weights which are calculated while generating that output?
One is internal (token prob), one is output generated by that internal mechanism e.g.,
"{ next_state: { "evade": 0.84, "land": 0.10, "search": 0.01"...}}"You have a ebnf grammar constrained to evade, land, search etc. Quite literally what's the difference between the log probs between those options and whatever number a very tiny model is coming up for the confidence. They're both literally the same concept.
Both use the softmax operation, but "probability of emitting a given label" is not the same as "probability the decided action is correct". OTOH, taking an action to sample something to come up with "probability of emitting a label" is in fact the whole shebang if the label is the next action. If it works it works.
Who cares, this distinction is small.
For me, a planning guy, either your suggestion or Jev-like would work just fine, and you better believe I'll be testing both as I'm quite excited about being able to use less-structured, untrained systems in decision pipelines. Thanks for the heads up.
Claude and GPT tell me all the time what they are confident and what they're not confident in. This is partially useful, but it's not a guarantee of anything, since they can be confidently wrong or unsure and correct.
Same with Jav.
Arguing that Jav in some way more useful than current LLMs because it includes a confidence score smells of someone who hasn't updated their priors since GPT-4/4o and it's particular issues.
Jav is cool because it's very fast, very cheap, and "good enough". That's it.
There are benchmarks for these things, which Jev and others have been put through; it's not a marketing boast!
* With jev-sec-bench the confidence numbers reached roughly the same as Qwen2.5-7b-instruct (confidence was off by about 6% on average in both cases).
* On the OpenRouter Banking 77 benchmark Jev did the worst out of all the models, being about 25 points off from reality.
I also think the fact that Jev hasn't published any real papers or actual technical details should make people a bit more skeptical than they have been.
If you don't care about that why wouldn't you the flagships??
I'm arguing against the idea that Jev fixes a fundamental deficiency in LLMs by showing it's confidence.
It doesn't.
Jev can do a strict subset of what LLMs can do, but incredibly quickly and incredibly cheaply, which is certainly valuable.
It is about getting calibrated outcomes. Jev is able to provide that, which is a major unlock
Maybe someone did order those enlargement pills, and an agentic model would be able to scan email history and determine that, when classifying an email as spam.
Jev's "confidence" scores are just as valid as an LLMs confidence score, i.e. just as likely to be wrong or hallucinated as any LLM.
Ultimately, people are somehow thinking Jev is mysteriously more likely to be right when ultimately it's another non-deterministic magic box except this time with a stricter interface slapped on top of it.
Are those scores accurate? Probably not. But are Jev's scores accurate? They certainly aren't with a torture test like "I've rolled a fair die. What is the result?" (It comes back with ~80% confidence on 1 in my testing.)
It seems like people are very enthusiastic about Jev giving scores for its results, but I'm not seeing a whole lot of attention paid to how good those scores are.
Interestingly, if I add a "unknown" option then my original question says unknown with 100% confidence. "I see several dots on top" gives 89% unknown. "The top is mostly white" gives 30% unknown and 37% 1.
So it definitely has some ability in this area, but it's iffy. This isn't exactly what it's made for but it doesn't seem too far off from the examples like taking in a customer email and deciding which team it belongs with.
* Latency: when you want to make a decision within 100ms
* Robotics: I want to try make a VLA version of it and see what happens
* Costs: For something that needs a yes/no answer every 10 seconds, it seems like it would be cheaper than an LLM API.
For a quick primer, Jev is able to provide confidence scores on its classification, i.e. it is able to calibrate how well it is able to predict. Being able to predict in a distribution is different from being able to calibrate confidence of the predictions which should happen from the question or domain distribution from which the decisions are predicted - being able to do that is tough and is not same as using LLMs logit probabilities which are predictions in the vocab space. Though both are loosely correlated and might converge as LLMs keep getting better, the former is a much stronger decision-making signal than the latter. Jev not beating LLM-as-a-judge might be due to various other reasons such as world knowledge etc, but Jev as a concept will always provide more reliable decisions / outputs than LLM-as-a-judge giving a scalar score.