That right there is the problem.
This is a scenario where the interest of the users and the labs actually align perfectly, since labs that have people actually paying for their product want benchmarks to align as close as possible to real life performance. Otherwise they can't actually easily tell whether the new model they are developing is any good or not, and so risk wasting literal billions of dollars on training runs for a product that people won't end up paying for. A failed major training run is an absolute disaster at the frontier, GPT-4.5 for instance was an almost existential failure for OpenAI at the time.
Case in point, the most popular coding model, Opus 5.5, also just happens to be the only one that avoided the benchmaxing impulse in this article. This is the result of years of Anthropic working very hard to avoid it.
DS used nearly twice the budget to accomplish the same thing.
Ideally, these tasks should be something you actually need day to day.
Astra does seem close.
I'd be curious to see Argon on there, since my experience is that Argon is also significantly behind Opus 5.5 and Astra (obviously much closer than Flash).
I've been meaning to take a closer look at what's behind the curtain of DeepSWE - since I built what I thought should be the easiest possible coding benchmark - and the models I tested it on scored roughly the same as they do on DeepSWE... Which is supposed to be hard...
Something seems amiss.
Especially since this benchmark is still mostly DeepSWE, and the only reason the open source models are doing decent is because much of the information hasn't changed. The open source models have been benchmaxxed to death.
Yes, this is true.
> You can not do real code work with open source models right now.
This is false, or you're using "real" as a loophole. You can absolutely do real coding work with open source models, right now, right this second.
I'm doing so today using Qwen 3.8 Flash Next running locally, and GLM 5.3 Flash hosted.
We could do real work with Opus 4.5, 4.6. These models (for coding) are more than good enough to do work that Opus 4.6 could do, and in some ways are superior.
uh-huh yeah, perhaps you can't
I only use open weights and get plenty of real work done. Turns out most real tasks are not that hard, rather they are tedious.
The practical takeaway, however, seems to be an interesting "don't use max mode". The passage about deepseeks weird API behavior dropping to pretty much zero the moment you used high says a lot.
Also, personally, even though Opus is clearly so much ahead of the Open Source models after the changes, I think the adjusted score is still very much competitive. I see other commenters using this as proof to justify why their opinions of the open source models - but a drop from 74% to ~50% is, while a big drop, still about a ~50% pass rate afterwards. That's far from useless for coding, especially since DeepSWE seems to be such a difficult benchmark. It simply seems to more accurately reflect the capabilities of these models.
All in all, I'd love to see more of such articles, and will probably keep an eye on this one - if they continue "benchmarking" these models this way.