Re:Senior SWE-Bench: open-source benchmark that assesses age...
by sebastiennight on Thursday July 02, 2026 @02:06PM
(#48765235)
> Is it giving an unfair advantage to Model X if we use Model X as the judge?
There have been studies that showed that models tended to rate responses from their own family of models better than equivalent responses from other families, eg. gpt-4 would prefer a response from gpt-3
Re:Senior SWE-Bench: open-source benchmark that assesses age...
by LiamPowell on Friday July 03, 2026 @01:08AM
(#48771005)
I suspected as much, and that brings us to the second issue where if we use a cohort of judges then the model that likes it's own code the most still wins.
Which LLM should we even use to judge taste? Is it giving an unfair advantage to Model X if we use Model X as the judge? Maybe we should use multiple models as the judge, but now the model that's best at recognising and praising its own code has an advantage. The whole thing is just an unsolvable problem when a LLM is the judge.