News for hackers. Stuff that matters.
Re:Senior SWE-Bench: open-source benchmark that assesses age... by sebastiennight on Thursday July 02, 2026 @02:06PM (#48765235)
> Is it giving an unfair advantage to Model X if we use Model X as the judge?

There have been studies that showed that models tended to rate responses from their own family of models better than equivalent responses from other families, eg. gpt-4 would prefer a response from gpt-3

Re:Senior SWE-Bench: open-source benchmark that assesses age... by LiamPowell on Friday July 03, 2026 @01:08AM (#48771005)
I suspected as much, and that brings us to the second issue where if we use a cohort of judges then the model that likes it's own code the most still wins.