News for hackers. Stuff that matters.
Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by esafak on Monday October 05, 2026 @12:05PM (#49966706)
That's a really weird way to test Jev. I don't see any mention of latency or cost in https://github.com/Gaurav-Gosain/jev-sec-bench

If you don't care about that why wouldn't you the flagships??

Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by exe34 on Monday October 05, 2026 @12:38PM (#49967137)
I think the point is if you can't get the right answer, it might not matter how fast you can make something up.
Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by esafak on Monday October 05, 2026 @12:53PM (#49967345)
It had an accuracy of 96.5% with context! Come on, guys.
Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by serbuvlad on Monday October 05, 2026 @01:43PM (#49967973)
I'm not arguing against Jev. Jev is very cool.

I'm arguing against the idea that Jev fixes a fundamental deficiency in LLMs by showing it's confidence.

It doesn't.

Jev can do a strict subset of what LLMs can do, but incredibly quickly and incredibly cheaply, which is certainly valuable.

Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by esafak on Monday October 05, 2026 @01:57PM (#49968162)
Yes, it does. A low-latency calibrated classifier is valuable.