MaxxBench – "Nice DeepSWE score. But is it true?"

Re:MaxxBench – "Nice DeepSWE score. But is it true?" by slopinthebag on Saturday November 07, 2026 @06:48PM (#50409148)

op's tests are underspecified one-shot tasks. it's entirely expected that smaller models perform poorly.

but your claim was about file name invariants when generating code, especially small amounts of code, which is not what they were doing with DeepSWE, so you've confused two different things.

anyways, this is a pointless argument. if you can't figure out how to effectively use smaller models in your workflow, fine. but don't come on here talking shit about people who can. have fun slopping out yet another ai harness at your startup.

[ Parent | on HN ]

Re:MaxxBench – "Nice DeepSWE score. But is it true?" by anonymous908213 on Saturday November 07, 2026 @06:53PM (#50409179)

While OP's tests are indeed of the vague DeepSWE variety, I still reference them because the exact same failure mode occurs in my own tests even for trivial scope tasks, down to the level of having one single constraint for a variety of <10 LoC tasks. As you say, though, this is a pointless argument, and I'm not going to burn my existing benchmarks to try to convince anyone.

[ Parent | on HN ]