News for hackers. Stuff that matters.
Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by wat10000 on Monday October 05, 2026 @11:12AM (#49965904)
LLMs can provide confidence scores. Just ask it and it will provide.

Are those scores accurate? Probably not. But are Jev's scores accurate? They certainly aren't with a torture test like "I've rolled a fair die. What is the result?" (It comes back with ~80% confidence on 1 in my testing.)

It seems like people are very enthusiastic about Jev giving scores for its results, but I'm not seeing a whole lot of attention paid to how good those scores are.

Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by wredcoll on Monday October 05, 2026 @11:51AM (#49966487)
That's an interesting test, but is it really what jev is supposed to be good at? I mean, this is a genuine question because I don't understand jev, but wouldn't it need a more constrained question?
Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by wat10000 on Monday October 05, 2026 @12:56PM (#49967378)
Probably not. But it does seem to have some idea of probabilities if I get a little more specific. "I've rolled a fair die. I see several dots on the top." Gives me 46% for 6, and about 16% for 4 and 5. If I say "The top is mostly white." then it says 1 with 50% confidence... and 6 with 31%.

Interestingly, if I add a "unknown" option then my original question says unknown with 100% confidence. "I see several dots on top" gives 89% unknown. "The top is mostly white" gives 30% unknown and 37% 1.

So it definitely has some ability in this area, but it's iffy. This isn't exactly what it's made for but it doesn't seem too far off from the examples like taking in a customer email and deciding which team it belongs with.

Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by wredcoll on Tuesday October 06, 2026 @04:17PM (#49983480)
This does feel like the sort of overly-confident but unfalsifiable output that llms excel at.