News for hackers. Stuff that matters.
Re:SlopCodeBench 10/2026 – Opus 5.5, Astra wins overall, xhi... by a3w on Tuesday October 20, 2026 @05:15PM (#50172783)
There are 36 tests. You can pass 0 or 36 or any number inbetween. Lets give five significant digits of how accurate solving was.

That is not how data physics work. It should not be how data science works.

Re:SlopCodeBench 10/2026 – Opus 5.5, Astra wins overall, xhi... by kirBurkhanov on Wednesday October 21, 2026 @05:48AM (#50178181)
Yeah, and the noise is also per model, run the same model five or ten times and put the confidence around it. Then try different models and different workflows. Thats actually how I do it when I build workflows and automations: test, test, and test, collect the statistic on where and what works better, then combine.