florianstandhar
florianstandhar on Hacker News
Re:JevBench, a reproducible benchmark for typed decision models
on Wednesday September 23, 2026 @07:32AM (#49814465)
Happy to — please open an issue at https://github.com/fstandhartinger/jevbench/issues with an endpoint or runnable code, the model and licence, and whether it was trained on the public items. Every entrant runs through the same harness, including the sealed set.
I definitely prefer open models I can run locally, because sending our private held-out set of tasks to an external API produces some headaches on ur side and basically means we have to rotate the test set a lot, to avoid contaimination.
We can call API hosted models though, we'll just flag the leaderboard entries appropriately.
Re:JevBench, a reproducible benchmark for typed decision models
on Wednesday September 23, 2026 @07:29AM (#49814443)
Thanks! Yeah, the code is open: https://github.com/fstandhartinger/who-is-right.
Email classifying is definitely a good Jev usecase, I agree
Re:JevBench, a reproducible benchmark for typed decision models
on Wednesday September 23, 2026 @07:26AM (#49814408)
Good call... BART-large-MNLI is the classic zero-shot baseline and it should be there. Adding it to the next run.
Re:OpenAI is well positioned to fast-follow Jev
on Tuesday September 22, 2026 @11:09AM (#49802610)
maybe open source even eats Jevs lunch first
Re:JevBench, a reproducible benchmark for typed decision models
on Tuesday September 22, 2026 @09:01AM (#49800575)
Hi HN!
I built JevBench because Jev kicks ass, and the world deserves to know how the serious open source and fake lookalike projects really perform in comparison.
Jev-class models return bounded choices and probabilities instead of text, and are disruptively faster and cheaper than LLMs, while being similarly intelligent on the text input they operate on.
JevBench allows looking at accuracy, latency and price all at once, in a weighted way - you can even configure the weighting.
A full run asks 534 English decisions. The v1.3 score combines chance-corrected Intelligence, Calibration, Speed and Cost.
Leaderboard right now:
#1 - Jev 74.4 #2 - SemIf 73.1 #3 - djev 73.0 #4 - Winnow-12B Q8 71.2 #5 reflex 4B 70.3.
MIT harness, public items, frozen artifacts, scoring code and public per-task outcomes: https://github.com/fstandhartinger/jevbench
Two no-signup demos: https://who-is-right.app.mintapis.com https://is-it-ai-slop.app.mintapis.com
Limitations: English-only; latency from one German server; local/demo latency gets a disclosed ×2 adjustment (+150 ms on my servers) which is an informed assumption; held-out prompts still reach evaluated services; ~1-point gaps can be noise.
Wdyt?