Re:FrontierCode
by tedsanders on Monday June 08, 2026 @07:37PM
(#48453976)
Makes sense, thanks. I suppose error bars are tricky if trying to handle problem-to-problem variance, rubric-to-rubric variance, and run-to-run variance all at once.
simple answer is our reporting was pass@5. feel like you'd need like 50+ runs to have reasonable confidence intervals, which somehow i dont see other people do, so i also didnt insist on it.
hoping to work with <prominent third party evals shop> to get this on their infra and evaluated along with whatever the industry standard is.