Re:In 27,280 coding agent runs, 64% of inference spend went ...
by visarga on Wednesday November 18, 2026 @12:00AM
(#50542074)
These kind of benchmarks declare "best harness" but measure just individual tasks with zero human interactivity, not project level efficiency. I don't do individual tasks, I work on larger projects made of 100s of tasks. The work across tasks, keeping intent alignment straight and testing reliable, remembering that detail from 5 days ago, that is also important.
This pretended equivalence is like claiming good LeetCode score means good engineering. I know it is much harder to measure project-wide performance but that is what we need.
This pretended equivalence is like claiming good LeetCode score means good engineering. I know it is much harder to measure project-wide performance but that is what we need.