Re:In 27,280 coding agent runs, 64% of inference spend went ...
by seizethecheese on Tuesday November 17, 2026 @12:25PM
(#50534870)
This doesn't answer the question at all.
Re:In 27,280 coding agent runs, 64% of inference spend went ...
by dotancohen on Tuesday November 17, 2026 @12:47PM
(#50535250)
It's still the most informative thing to come out of that account or the linked article.
Re:In 27,280 coding agent runs, 64% of inference spend went ...
by numeri on Tuesday November 17, 2026 @06:33PM
(#50539751)
unless they edited it in later, the link at the bottom lists which tasks they ran it on. It definitely doesn't answer why those tasks or why specifically 62 of them
Re:In 27,280 coding agent runs, 64% of inference spend went ...
by Edward40 on Tuesday November 17, 2026 @06:39PM
(#50539822)
We just want to keep the category unified so we have tasks for every domain like ML, science and others, and also have a chance to see the divergence from tb2.1 to tb4.
Re:In 27,280 coding agent runs, 64% of inference spend went ...
by visarga on Wednesday November 18, 2026 @12:00AM
(#50542074)
These kind of benchmarks declare "best harness" but measure just individual tasks with zero human interactivity, not project level efficiency. I don't do individual tasks, I work on larger projects made of 100s of tasks. The work across tasks, keeping intent alignment straight and testing reliable, remembering that detail from 5 days ago, that is also important.
This pretended equivalence is like claiming good LeetCode score means good engineering. I know it is much harder to measure project-wide performance but that is what we need.
For scoring, final passes are divided by scored attempts, where infrastructure errors are excluded from the denominator, timeouts are still graded by the verifier, and reward hacking (such as looking up solutions online) counts as a failure. Costs are calculated using list price multiplied by reported tokens.
https://hub.harborframework.com/datasets/frontier-harness-ev...