News for hackers. Stuff that matters.
Re:Claude Opus 5 by 2001zhaozhao on Friday July 24, 2026 @02:26PM (#49039743)
It is probably from randomness. The benchmark tasks nowadays are so long that you can't really afford to run a large number of samples of them per model & effort combination
Re:Claude Opus 5 by artemisart on Friday July 24, 2026 @03:39PM (#49040678)
No it's a mean of 5 runs.

> We report FrontierCode’s overall score, a composite measure that grades each patch on blocking functional criteria (held-out unit tests) together with weighted code-quality rubric criteria, as mean@5.

They don't explain more in the system card, I guess higher effort levels could loose points on the code quality / scope / style / maintainability stuff?

Re:Claude Opus 5 by dannyw on Saturday July 25, 2026 @02:21AM (#49044991)
If you're doing passes@1, especially for long-horizon agentic benchmarks, you might as well as not do the benchmark at all.