Re:MaxxBench – "Nice DeepSWE score. But is it true?"
by andai on Saturday November 07, 2026 @02:27PM
(#50407243)
> Most models tried pleasing an imagined grader over the prompt
>When we read the trajectories, we saw that many models had figured out they were in a benchmark and started second-guessing the prompt. Instead of just building what we asked for, they started coding to the tests they thought the grader would run.
>When we read the trajectories, we saw that many models had figured out they were in a benchmark and started second-guessing the prompt. Instead of just building what we asked for, they started coding to the tests they thought the grader would run.
I ran into the same problem on my English exams.