News for hackers. Stuff that matters.
Re:Decision models like Jev don't beat LLM-as-a-judge or tra... by bunderbunder on Monday October 05, 2026 @09:37AM (#49964727)
Usably good calibration is hard. It's hard with logistic regression, it's even harder with linear support vector machines, and it makes me question my life choices with non-linear models.

It's not necessarily because the model is wrong. I've had trouble getting useful calibration out of models with an F1 of 0.9. The fundamental problem is twofold. First, it turns out that [0.0, 1.0] is a much larger set than {0, 1}. Second, the kinds of use cases where you care about calibration tend to be fussy and demanding.

As a fun anecdote, I got a chance to ask the person who invented the method scikit-learn uses for calibrated SVMs if he had any advice, and his answer was basically, "good luck."

All that said, sometimes you don't actually need calibration; you just need decent ranking. "Items with a score of 0.9 should be more likely to be positive than ones with a score of 0.5," is an easier requirement than "90% of items with a score of 0.9 should be positive." But I've had trouble getting that out of highly non-linear neural models, too, because they oftentimes produce results where the relationship between score and probability of being in the positive class is not even remotely monotonic.

But I haven't poked at Jev like this either, so I do have to allow that maybe they've found the secret sauce.

There are no comments on this comment yet.