A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.
And even if they did try to test them on a lot of gotcha tasks; I'm perennially reminded of the ever-relevant quote about AI and other tech: "this is the worst it will ever be"
My Windows work desktop has two orders of magnitude more memory and more processing power than the last time I used Windows, yet boots slower and is laggier than any Windows I've ever used.
Almost all useful information online is concentrated into YouTube and Reddit, everything is advertising and trying to steal attention. I'm not even welcome on Reddit anymore due to the media campaigns against my country and race.
We've definitely passed the point where the derivative has gone negative.
We kept the category unified so we have tasks for every domain like ML , science and others.
For scoring, final passes are divided by scored attempts, where infrastructure errors are excluded from the denominator, timeouts are still graded by the verifier, and reward hacking (such as looking up solutions online) counts as a failure. Costs are calculated using list price multiplied by reported tokens.
https://hub.harborframework.com/datasets/frontier-harness-ev...
This pretended equivalence is like claiming good LeetCode score means good engineering. I know it is much harder to measure project-wide performance but that is what we need.
For harness developers: cut steps, not just prompt tokens. Prompt size, tool count and cache hit rate did not predict cost per pass. Steps did, because every step sends the whole context again.
Incredible, none of these harness developers considered just asking the LLM to solve the problem right away! Don't siege Leningrad, take it immediately!
Then, somebody runs a small subset of benchmarks - with not enough data to have any statistical significance - and makes further claims.
How are we supposed to derive value from this?
It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.