Re:MaxxBench – "Nice DeepSWE score. But is it true?"
by Topfi on Sunday November 08, 2026 @09:36AM
(#50413728)
Great job, this mirrors my experience running private evals nearly 1:1, down to even GPT models post Spud pre-train (so anything starting with GPT-5.5 to now) underperforming what one would expect based on public evals, the second one runs them privately. Also, Anthropic models have never under and often over-performed in my evals focused on task adherence, task evaluation, task inference and tool calling.
Love having a clear, concise and actually based in tangible information post to point to next time someone tells me that me models like Deepseek, Qwen and Grok massively underperforming what the public numbers show means I must simply not be capable enough to use them properly, even when it comes to tasks that Opus 4.5 managed but deviate from what most public evals have.
Hope that this will finally get certain people to accept that there is massive training to the test by certain labs and that labs like Anthropic have shown for years that, no, this isn't inevitable and one can build models that are not just able to match but even outperform private evals. Same thing with the Pelican SVG, shared examples often enough of that being "benchmaxxed" by Qwen and Grok, which perform poor on similar requests without a bike or pelican. Not sure whether it is more troubling that some labs are so insecure as to do this or that we, despite the evidence, still act like this isn't the case, but that's life.
Love having a clear, concise and actually based in tangible information post to point to next time someone tells me that me models like Deepseek, Qwen and Grok massively underperforming what the public numbers show means I must simply not be capable enough to use them properly, even when it comes to tasks that Opus 4.5 managed but deviate from what most public evals have.
Hope that this will finally get certain people to accept that there is massive training to the test by certain labs and that labs like Anthropic have shown for years that, no, this isn't inevitable and one can build models that are not just able to match but even outperform private evals. Same thing with the Pelican SVG, shared examples often enough of that being "benchmaxxed" by Qwen and Grok, which perform poor on similar requests without a bike or pelican. Not sure whether it is more troubling that some labs are so insecure as to do this or that we, despite the evidence, still act like this isn't the case, but that's life.