Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

Re:Show HN: FrontierHarness Eval – 9 harness, same model, co... by nsingh2 on Wednesday September 02, 2026 @03:15PM (#49541015)

One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup.

Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.

[ Parent | on HN ]

Re:Show HN: FrontierHarness Eval – 9 harness, same model, co... by dfltr on Wednesday September 02, 2026 @03:45PM (#49541405)

It does seem a bit odd to say "We tested all of these bicycles with the same rider" when one of the test cases is actually a bare high-end frame with no components on it.

It's even weirder given that another of the test cases is essentially "We put parts on the frame (OhMyPi) and it went faster!"

[ Parent | on HN ]

Re:Show HN: FrontierHarness Eval – 9 harness, same model, co... by jdthedisciple on Wednesday September 02, 2026 @05:05PM (#49542559)

I'm not sure your analogy holds, because here the optimization metric is clear and unanimous: every one wants max pass rate at min costs.

With the bicycle, some may prefer comfort, others speed, others offroad, etc., so it would not be obvious which one is "best".

[ Parent | on HN ]

Re:Show HN: FrontierHarness Eval – 9 harness, same model, co... by infecto on Wednesday September 02, 2026 @05:16PM (#49542688)

Not weird or odd in the least. I think it clearly shows that Pi barebones at least in this set of tests, preformed better and cheaper cost that OMP.

[ Parent | on HN ]

Re:Show HN: FrontierHarness Eval – 9 harness, same model, co... by randomblock1 on Wednesday September 02, 2026 @03:45PM (#49541413)

I think it's still a useful data point. For example, omp, which is pi with some default extensions, scores worse. I do agree that adding more configurations of Pi would help though.

[ Parent | on HN ]

Re:Show HN: FrontierHarness Eval – 9 harness, same model, co... by infecto on Wednesday September 02, 2026 @05:16PM (#49542695)

I use Pi barebones.

[ Parent | on HN ]

Re:Show HN: FrontierHarness Eval – 9 harness, same model, co... by Edward40 on Wednesday September 02, 2026 @07:00PM (#49543761)

I also prefer using vanilla Pi over Oh My Pi.

[ Parent | on HN ]