News for hackers. Stuff that matters.

A comment on Top model scores may be skewed by Git history leaks in SWE-bench. Parent comment

Re:Top model scores may be skewed by Git history leaks in SW... by tomjakubowski on Thursday September 11, 2025 @05:40PM (#45216382)
Why do you doubt that? It's a widely used language. And there is even an open source C# REPL.
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by CuriouslyC on Thursday September 11, 2025 @06:16PM (#45216670)
Because RL time is expensive and I don't think the languages which are more popular than C# have such high performance that it's worth bumping their batches for C#.
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by stingraycharles on Thursday September 11, 2025 @07:37PM (#45217220)
But C# is a typical enterprise language which has people who are willing to pay a lot of money for AI.

We’re just guessing and the fact of the matter is that we don’t know what inputs they use for their models.

[ Parent | on HN ]