News for hackers. Stuff that matters.
Re:Jev in 25 Lines of Python by sidclaw on Thursday September 24, 2026 @04:16PM (#49836250)
Thoughtful demo. If possible, I'd want to see next is whether the probabilities it returns are calibrated and not just whether the top choice is right. I ran a small experiment on this, compared base and instruct models' calibration on option-letter logits, then fine-tuned with Brier loss vs cross-entropy. Have you looked at a reliability curve for this version?

There are no comments on this comment yet.