News for hackers. Stuff that matters.
Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by jvanderbot on Monday October 05, 2026 @08:23AM (#49963921)
Partially so. But the fact that we're comparing Jev to SOA LLM and pre-trained classification means it's essentially satisfied its reason for existence.

The benefits of Jev, and the reason I'm excited about them:

* Pre-trained/tuned to provide only structured output

* Small, fast, essentially trivial to locally run - this alone makes them an actual candidate for real-world planning systems. Laya is a local-first 400 million parameter model, for example.

* Their confidence scores are useful, even if those scores are not correct/true probabilities.

For me and my work, they look like nearly turn-key, tiny, fast, locally-hostable models that can manage state transitions in a deeply autonomous system.

LLM-as judge requires, well, what we know as a full LLM. A fully trained classifier model requires fully training - something you cannot / won't do for a embedded autonomous planning system (by contradiction - if this worked, we'd have used it everywhere already!). Totally unfair comparison in my mind.

Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by monocasa on Monday October 05, 2026 @09:55AM (#49964960)
With llama.cpp, you can give it an ebnf grammar and constrain the output to any formal grammar you wish.

And how is the confidence score any different than the softmaxed token probabilities you get out of running every llm?

Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by ekidd on Monday October 05, 2026 @11:14AM (#49965929)
> With llama.cpp, you can give it an ebnf grammar and constrain the output to any formal grammar you wish.

With llama-server, you can use basically any GGUF model as a Jev-like classifier. See Pi.dev's codemode and the accompanying llama backend to the "classify" function. The advantages of Jev-like inference are that:

- You generate 1 token per question in parallel, instead of sequentially generating a bunch of JSON punctuation. So you get much better latency and utilization.

- You can use a custom "sampler" that sees the raw logprobs for these tokens, including all the tokens that might have been generated. So you can generate some "probability" or "confidence" numbers based on how likely the model was to have generated a different answer.

So really, no difference? It's just a nice API with some computational efficiencies.

Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by S0y on Monday October 05, 2026 @11:55AM (#49966564)
ebnf grammar is a truly underrated feature of llamacpp.
Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by jvanderbot on Monday October 05, 2026 @10:19AM (#49965245)
> And how is the confidence score any different than the softmaxed token probabilities you get out of running every llm?

You're asking me how a structured output which assigns probabilities to classifications of the input text is different than the next-token weights which are calculated while generating that output?

One is internal (token prob), one is output generated by that internal mechanism e.g.,

    "{ next_state: { "evade": 0.84, "land": 0.10, "search": 0.01"...}}"
Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by monocasa on Monday October 05, 2026 @12:10PM (#49966758)
Yes. I am.

You have a ebnf grammar constrained to evade, land, search etc. Quite literally what's the difference between the log probs between those options and whatever number a very tiny model is coming up for the confidence. They're both literally the same concept.

Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by jvanderbot on Monday October 05, 2026 @01:16PM (#49967637)
Copy. The mechanism is probably similar once we abandon the "LLM that is asked to generate json" path. But the grammar alone doesn't ensure the output is a probability vector does it?

Both use the softmax operation, but "probability of emitting a given label" is not the same as "probability the decided action is correct". OTOH, taking an action to sample something to come up with "probability of emitting a label" is in fact the whole shebang if the label is the next action. If it works it works.

Who cares, this distinction is small.

For me, a planning guy, either your suggestion or Jev-like would work just fine, and you better believe I'll be testing both as I'm quite excited about being able to use less-structured, untrained systems in decision pipelines. Thanks for the heads up.

Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by vrighter on Tuesday October 06, 2026 @05:40AM (#49976223)
the "provide only structured output" is done by NOT having the model do it, but regular old software. So tuning/training doesn't apply for that.