News for hackers. Stuff that matters.
Re:MaxxBench – "Nice DeepSWE score. But is it true?" by rao-v on Saturday November 07, 2026 @06:20PM (#50408991)
My understanding is this is less about labs training on the benchmark as being lazy about creating RL environments based on the benchmark.

They should be doing what this test did and generalizing and significantly varying the training environments (and ensuring some of them are unexpectedly complex and unexpectedly simple). The smaller models are just overfitting to the simpler RL envs because that is the arena where they generally get to the 30-50% success rate you want for training

Re:MaxxBench – "Nice DeepSWE score. But is it true?" by Muromec on Saturday November 07, 2026 @08:53PM (#50409817)
Right, how can anything that mentions grading, not even mentioning actively cheating, is allowed out of the door.

I care not so much about wrong answer, that's just another try and it gets it right eventually, but this cheating.