Edward40

I am really impressed by Runta's infra. Integrating remote desktop was basically effortless.

Wild that we built an open-source Grok Bot alternative in just one week on top of Runta. Remote desktop integration was practically plug-and-play.

Claude Code is an opinionated harness. I’d love to see whether neutral harnesses can challenge the monolithic harness narrative.

We'd rather not mess with system prompts, we just eval them as shipped to keep the results reproducible.

I also prefer using vanilla Pi over Oh My Pi.

Good points! That’s a known limitation of our v1.0 benchmark with Claude Code. For v1.1, we're expanding both harnesses and models to evaluate the full harness × model matrix. This will highlight interaction effects (showing which harnesses suit which models best) and let us publish a full compatibility matrix. We shared more details on our roadmap in our blog post (https://runta.com/blog/introducing-frontierharness-eval/).

We initially tried using the mean (average), but a few extreme outliers caused Claude Code's cost to look far higher than it typically is.

Version 1.1 will add more harnesses and evaluate a broader set of models. Rather than holding the model fixed, we plan to evaluate the full harness × model matrix to reveal interaction effects (which harnesses work best with which models) and produce a harness-model compatibility matrix.

The Exo harness is the most interesting one in the benchmark, as it has the potential to complete tasks with fewer turns and cost.