Re:Decision models like Jev don't beat LLM-as-a-judge or tra...
by wredcoll on Monday October 05, 2026 @11:51AM
(#49966487)
That's an interesting test, but is it really what jev is supposed to be good at? I mean, this is a genuine question because I don't understand jev, but wouldn't it need a more constrained question?
Re:Decision models like Jev don't beat LLM-as-a-judge or tra...
by wat10000 on Monday October 05, 2026 @12:56PM
(#49967378)
Probably not. But it does seem to have some idea of probabilities if I get a little more specific. "I've rolled a fair die. I see several dots on the top." Gives me 46% for 6, and about 16% for 4 and 5. If I say "The top is mostly white." then it says 1 with 50% confidence... and 6 with 31%.
Interestingly, if I add a "unknown" option then my original question says unknown with 100% confidence. "I see several dots on top" gives 89% unknown. "The top is mostly white" gives 30% unknown and 37% 1.
So it definitely has some ability in this area, but it's iffy. This isn't exactly what it's made for but it doesn't seem too far off from the examples like taking in a customer email and deciding which team it belongs with.
Are those scores accurate? Probably not. But are Jev's scores accurate? They certainly aren't with a torture test like "I've rolled a fair die. What is the result?" (It comes back with ~80% confidence on 1 in my testing.)
It seems like people are very enthusiastic about Jev giving scores for its results, but I'm not seeing a whole lot of attention paid to how good those scores are.