News for hackers. Stuff that matters.

A comment on Top model scores may be skewed by Git history leaks in SWE-bench. Parent comment

Re:Top model scores may be skewed by Git history leaks in SW... by fine_tune on Thursday September 11, 2025 @03:02PM (#45214975)
I was going to argue "LLM's need code samples to-do well on languages and if we are honest C# is a language mostly held in private repo's" but Github's 2024 report[0] says its the 5th most used language (I'm to lazy to check if this report includes private repo's but I'll assume it doesn't).

So kinda neat to see this paper!

[0]https://github.blog/news-insights/octoverse/octoverse-2024/#...

[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by yieldcrv on Thursday September 11, 2025 @03:34PM (#45215307)
5th most used language based on private repos that the group making the report has the exclusive direct access to seeing

I don't see that contradicting your assumption

[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by BoorishBears on Thursday September 11, 2025 @04:30PM (#45215817)
"In this year’s Octoverse report, we study how public and open source activity on GitHub..."
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by CuriouslyC on Thursday September 11, 2025 @04:41PM (#45215922)
The big labs are almost certainly using compiler/repl output for generated code as an oracle for RL. I doubt they have C# in the mix.
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by tomjakubowski on Thursday September 11, 2025 @05:40PM (#45216382)
Why do you doubt that? It's a widely used language. And there is even an open source C# REPL.
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by CuriouslyC on Thursday September 11, 2025 @06:16PM (#45216670)
Because RL time is expensive and I don't think the languages which are more popular than C# have such high performance that it's worth bumping their batches for C#.
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by stingraycharles on Thursday September 11, 2025 @07:37PM (#45217220)
But C# is a typical enterprise language which has people who are willing to pay a lot of money for AI.

We’re just guessing and the fact of the matter is that we don’t know what inputs they use for their models.

[ Parent | on HN ]