Re:Clef: Open-source decision models, and new RL fine-tuning...
by segmondy on Thursday October 01, 2026 @05:37PM
(#49927360)
A lot of people claim to have made better than jev, there's a jev benchmark, I have tried many of those models and they eventually end up failing, a non trivial task which doesn't seem like much but reminds me of the svg pelican bench is games, have one of these decision/classifier models play a game, hook it up to the input, most of the ones that are supposedly on jev level end up playing a terrible game, showing that they are very narrow. Cloudflare doesn't compare to the top open bench alternatives, I just finished downloading it and will compare it to jev for non trivial tasks tonight.
[ Parent
| on HN ]
Re:Clef: Open-source decision models, and new RL fine-tuning...
by SebastianSosa on Thursday October 01, 2026 @06:47PM
(#49927950)
Public benchmarks are easy to cheat, if I am typesafe I would also release a public benchmark to distract otherwise competent people in overfitting to a benchmark instead of making something actually useful.
Diogo very much is against public benchmarks ;)
Re:Clef: Open-source decision models, and new RL fine-tuning...
by Foobar8568 on Friday October 02, 2026 @08:08AM
(#49932578)
You take Qwen3.6 35b on a 5090rtx, and here you get a higher score than Jev, for 2sec more latency on average.
So yeah it's not subsecond, but I am sure that if I had VC money, I could too get within 500ms too!
Re:Clef: Open-source decision models, and new RL fine-tuning...
by verdverm on Thursday October 01, 2026 @06:49PM
(#49927964)
watching Jev play Pokemon demonstrated this too, more hype than meat
- I'd like a potion, are you sure, no, repeat
- in and out of doors on loop
- sisyphean effort in the cave
- jev-ish level grinding
It was impressive, beat pokemon for less than $2, but not all that interesting. People asking how different Math.Random plays pokemon would be, and at the other end, regular llms playing games.
Re:Clef: Open-source decision models, and new RL fine-tuning...
by indoor47 on Friday October 02, 2026 @07:19AM
(#49932234)
Well, it makes sense:
"Clef builds upon this concept, but uses a different base model as the backbone. We currently use Qwen as the base model and post-trained it to suit decision model use cases. "