MaxxBench – "Nice DeepSWE score. But is it true?"

Re:MaxxBench – "Nice DeepSWE score. But is it true?" by onlyrealcuzzo on Saturday November 07, 2026 @04:34PM (#50408270)

This DEFINITELY squares with my anecdata that Gemini 3.8 Flash is not even remotely close to Opus 5.5 on the types of problems DeepSWE CLAIMS to be benchmarking.

Astra does seem close.

I'd be curious to see Argon on there, since my experience is that Argon is also significantly behind Opus 5.5 and Astra (obviously much closer than Flash).

I've been meaning to take a closer look at what's behind the curtain of DeepSWE - since I built what I thought should be the easiest possible coding benchmark - and the models I tested it on scored roughly the same as they do on DeepSWE... Which is supposed to be hard...

Something seems amiss.

[ Parent | on HN ]

Re:MaxxBench – "Nice DeepSWE score. But is it true?" by jtwaleson on Sunday November 08, 2026 @03:31AM (#50411429)

We recently benchmarked our AI code review agent (just pi with some custom prompts based on info from our context layer). Previously we used Gemini flash 3.8 as it was capable and affordable. However, haiku 5.5 just blows it out of the water. It's more than 10x cheaper, 2x faster and finds 3x as many real issues.

[ Parent | on HN ]