News for hackers. Stuff that matters.
Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by jvanderbot on Monday October 05, 2026 @10:19AM (#49965245)
> And how is the confidence score any different than the softmaxed token probabilities you get out of running every llm?

You're asking me how a structured output which assigns probabilities to classifications of the input text is different than the next-token weights which are calculated while generating that output?

One is internal (token prob), one is output generated by that internal mechanism e.g.,

    "{ next_state: { "evade": 0.84, "land": 0.10, "search": 0.01"...}}"
Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by monocasa on Monday October 05, 2026 @12:10PM (#49966758)
Yes. I am.

You have a ebnf grammar constrained to evade, land, search etc. Quite literally what's the difference between the log probs between those options and whatever number a very tiny model is coming up for the confidence. They're both literally the same concept.

Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by jvanderbot on Monday October 05, 2026 @01:16PM (#49967637)
Copy. The mechanism is probably similar once we abandon the "LLM that is asked to generate json" path. But the grammar alone doesn't ensure the output is a probability vector does it?

Both use the softmax operation, but "probability of emitting a given label" is not the same as "probability the decided action is correct". OTOH, taking an action to sample something to come up with "probability of emitting a label" is in fact the whole shebang if the label is the next action. If it works it works.

Who cares, this distinction is small.

For me, a planning guy, either your suggestion or Jev-like would work just fine, and you better believe I'll be testing both as I'm quite excited about being able to use less-structured, untrained systems in decision pipelines. Thanks for the heads up.