Re:Decision models like Jev don't beat LLM-as-a-judge or tra...
by tedivm on Monday October 05, 2026 @11:52AM
(#49966511)
The benchmarks show it is on par or worse than even older LLM models.
* With jev-sec-bench the confidence numbers reached roughly the same as Qwen2.5-7b-instruct (confidence was off by about 6% on average in both cases).
* On the OpenRouter Banking 77 benchmark Jev did the worst out of all the models, being about 25 points off from reality.
I also think the fact that Jev hasn't published any real papers or actual technical details should make people a bit more skeptical than they have been.
Re:Decision models like Jev don't beat LLM-as-a-judge or tra...
by esafak on Monday October 05, 2026 @12:05PM
(#49966706)
That's a really weird way to test Jev. I don't see any mention of latency or cost in https://github.com/Gaurav-Gosain/jev-sec-bench
If you don't care about that why wouldn't you the flagships??
Re:Decision models like Jev don't beat LLM-as-a-judge or tra...
by exe34 on Monday October 05, 2026 @12:38PM
(#49967137)
I think the point is if you can't get the right answer, it might not matter how fast you can make something up.
Re:Decision models like Jev don't beat LLM-as-a-judge or tra...
by esafak on Monday October 05, 2026 @12:53PM
(#49967345)
It had an accuracy of 96.5% with context! Come on, guys.
Re:Decision models like Jev don't beat LLM-as-a-judge or tra...
by serbuvlad on Monday October 05, 2026 @01:43PM
(#49967973)
I'm not arguing against Jev. Jev is very cool.
I'm arguing against the idea that Jev fixes a fundamental deficiency in LLMs by showing it's confidence.
It doesn't.
Jev can do a strict subset of what LLMs can do, but incredibly quickly and incredibly cheaply, which is certainly valuable.
There are benchmarks for these things, which Jev and others have been put through; it's not a marketing boast!