News for hackers. Stuff that matters.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by seizethecheese on Tuesday November 17, 2026 @11:36AM (#50533954)
The headline here is confusing. They ran a bunch of harness and agent combinations on terminal bench tasks, and 64% of the spending was on tasks that didn’t pass terminal bench verifiers.

It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.

Re:In 27,280 coding agent runs, 64% of inference spend went ... by seizethecheese on Tuesday November 17, 2026 @11:45AM (#50534140)
Okay, I wanted to know “which tasks” and dig into the methodology, and its documented nowhere!

A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.

Re:In 27,280 coding agent runs, 64% of inference spend went ... by jzemeocala on Tuesday November 17, 2026 @11:57AM (#50534379)
For real, i remember setting up advanced Linux configurations back before the 3.0 kernel and I'm pretty sure I had a higher failure rate.

And even if they did try to test them on a lot of gotcha tasks; I'm perennially reminded of the ever-relevant quote about AI and other tech: "this is the worst it will ever be"

Re:In 27,280 coding agent runs, 64% of inference spend went ... by dotancohen on Tuesday November 17, 2026 @12:55PM (#50535378)
That quote is outdated. My phone nags me now twice daily to update, and the update removes features I use so I'm careful to tread around it.

My Windows work desktop has two orders of magnitude more memory and more processing power than the last time I used Windows, yet boots slower and is laggier than any Windows I've ever used.

Almost all useful information online is concentrated into YouTube and Reddit, everything is advertising and trying to steal attention. I'm not even welcome on Reddit anymore due to the media campaigns against my country and race.

We've definitely passed the point where the derivative has gone negative.

Re:In 27,280 coding agent runs, 64% of inference spend went ... by seizethecheese on Tuesday November 17, 2026 @12:00PM (#50534425)
Confirmed. This is 17 TB4 tasks and 45 TB2.1 tasks. Why?
Re:In 27,280 coding agent runs, 64% of inference spend went ... by Edward40 on Tuesday November 17, 2026 @12:03PM (#50534473)
We want to see how performance diverges between TB2.1 and TB4 while keeping costs manageable, since running the entire dataset is really expensive.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by seizethecheese on Tuesday November 17, 2026 @12:07PM (#50534538)
How were the specific questions chosen?
Re:In 27,280 coding agent runs, 64% of inference spend went ... by Edward40 on Tuesday November 17, 2026 @01:07PM (#50535581)
You can see the task in https://hub.harborframework.com/datasets/frontier-harness-ev...

We kept the category unified so we have tasks for every domain like ML , science and others.

Re:In 27,280 coding agent runs, 64% of inference spend went ... by seizethecheese on Tuesday November 17, 2026 @01:44PM (#50536192)
[delayed]
Re:In 27,280 coding agent runs, 64% of inference spend went ... by Edward40 on Tuesday November 17, 2026 @12:19PM (#50534759)
Sorry for the inconvenience. We ran 27,280 total trials on Runta (1–4 vCPU, 2–8 GB RAM, 32 GB disk) using the 62-task [email protected] dataset across 5 models and 22 harnesses on Baseten, with 4 attempts per task. Every harness and model ran at max effort with a 131,072 max output token limit per request and per-task agent timeouts ranging from 15 minutes to 8 hours.

For scoring, final passes are divided by scored attempts, where infrastructure errors are excluded from the denominator, timeouts are still graded by the verifier, and reward hacking (such as looking up solutions online) counts as a failure. Costs are calculated using list price multiplied by reported tokens.

https://hub.harborframework.com/datasets/frontier-harness-ev...

Re:In 27,280 coding agent runs, 64% of inference spend went ... by seizethecheese on Tuesday November 17, 2026 @12:25PM (#50534870)
This doesn't answer the question at all.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by dotancohen on Tuesday November 17, 2026 @12:47PM (#50535250)
It's still the most informative thing to come out of that account or the linked article.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by numeri on Tuesday November 17, 2026 @06:33PM (#50539751)
unless they edited it in later, the link at the bottom lists which tasks they ran it on. It definitely doesn't answer why those tasks or why specifically 62 of them
Re:In 27,280 coding agent runs, 64% of inference spend went ... by Edward40 on Tuesday November 17, 2026 @06:39PM (#50539822)
We just want to keep the category unified so we have tasks for every domain like ML, science and others, and also have a chance to see the divergence from tb2.1 to tb4.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by visarga on Wednesday November 18, 2026 @12:00AM (#50542074)
These kind of benchmarks declare "best harness" but measure just individual tasks with zero human interactivity, not project level efficiency. I don't do individual tasks, I work on larger projects made of 100s of tasks. The work across tasks, keeping intent alignment straight and testing reliable, remembering that detail from 5 days ago, that is also important.

This pretended equivalence is like claiming good LeetCode score means good engineering. I know it is much harder to measure project-wide performance but that is what we need.

Re:In 27,280 coding agent runs, 64% of inference spend went ... by Edward40 on Tuesday November 17, 2026 @12:20PM (#50534783)
I'm also building a leaderboard in the meantime, which will release in a few days with more detailed harness version info and methodology.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by plufz on Tuesday November 17, 2026 @11:46AM (#50534177)
Yeah exactly, compared to what baseline? People here will probably be stunned when I say this, but my pre-agent code was usually not bug free on my first attempts either.
[dead]
Re:In 27,280 coding agent runs, 64% of inference spend went ... by warkdarrior on Tuesday November 17, 2026 @01:23PM (#50535858)
I wouldn't call it "burning of tokens and compute" if you get a useful signal out of it.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by themgt on Tuesday November 17, 2026 @11:59AM (#50534405)
Over the 22 harnesses, the rank correlation of the steps per pass and the cost per pass was +0.51 (p = 0.015). For the other three factors, it was −0.03 to +0.02 (p ≥ 0.88). Within each model, tool calls tracked cost as well as steps. Table 4 below and Table 5 in the appendix give the values.

For harness developers: cut steps, not just prompt tokens. Prompt size, tool count and cache hit rate did not predict cost per pass. Steps did, because every step sends the whole context again.

Incredible, none of these harness developers considered just asking the LLM to solve the problem right away! Don't siege Leningrad, take it immediately!

Re:In 27,280 coding agent runs, 64% of inference spend went ... by Edward40 on Tuesday November 17, 2026 @12:04PM (#50534490)
Cutting the system prompt and tools is easy. Cutting steps requires understanding the model's RL strategy and how it was trained to explore.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by bisonbear on Tuesday November 17, 2026 @12:17PM (#50534736)
I wonder if Claude Code would do better given they just introduced code mode, which should significantly cut the number of steps required
Re:In 27,280 coding agent runs, 64% of inference spend went ... by adrithmetiqa on Tuesday November 17, 2026 @12:46PM (#50535232)
Agreed. Does this help anyone decide what model or harness to use. Nope.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by Edward40 on Tuesday November 17, 2026 @01:08PM (#50535603)
Good finding! Model preference is interesting.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by onlyrealcuzzo on Tuesday November 17, 2026 @03:08PM (#50537409)
There's thousands of benchmarks, and nobody can agree on what the best models are if you run all of the benchmarks which costs tens of thousands of dollars for frontier models.

Then, somebody runs a small subset of benchmarks - with not enough data to have any statistical significance - and makes further claims.

How are we supposed to derive value from this?

Re:In 27,280 coding agent runs, 64% of inference spend went ... by Edward40 on Tuesday November 17, 2026 @04:15PM (#50538298)
Although people disagree with the best models, the worst model is obvious.