News for hackers. Stuff that matters.
Re:Clef: Open-source decision models, and new RL fine-tuning... by segmondy on Thursday October 01, 2026 @05:37PM (#49927360)
A lot of people claim to have made better than jev, there's a jev benchmark, I have tried many of those models and they eventually end up failing, a non trivial task which doesn't seem like much but reminds me of the svg pelican bench is games, have one of these decision/classifier models play a game, hook it up to the input, most of the ones that are supposedly on jev level end up playing a terrible game, showing that they are very narrow. Cloudflare doesn't compare to the top open bench alternatives, I just finished downloading it and will compare it to jev for non trivial tasks tonight.
Re:Clef: Open-source decision models, and new RL fine-tuning... by SebastianSosa on Thursday October 01, 2026 @06:47PM (#49927950)
Public benchmarks are easy to cheat, if I am typesafe I would also release a public benchmark to distract otherwise competent people in overfitting to a benchmark instead of making something actually useful. Diogo very much is against public benchmarks ;)
Re:Clef: Open-source decision models, and new RL fine-tuning... by Foobar8568 on Friday October 02, 2026 @08:08AM (#49932578)
You take Qwen3.6 35b on a 5090rtx, and here you get a higher score than Jev, for 2sec more latency on average. So yeah it's not subsecond, but I am sure that if I had VC money, I could too get within 500ms too!
Re:Clef: Open-source decision models, and new RL fine-tuning... by verdverm on Thursday October 01, 2026 @06:49PM (#49927964)
watching Jev play Pokemon demonstrated this too, more hype than meat

- I'd like a potion, are you sure, no, repeat

- in and out of doors on loop

- sisyphean effort in the cave

- jev-ish level grinding

It was impressive, beat pokemon for less than $2, but not all that interesting. People asking how different Math.Random plays pokemon would be, and at the other end, regular llms playing games.

Re:Clef: Open-source decision models, and new RL fine-tuning... by indoor47 on Friday October 02, 2026 @07:19AM (#49932234)
Well, it makes sense:

"Clef builds upon this concept, but uses a different base model as the backbone. We currently use Qwen as the base model and post-trained it to suit decision model use cases. "