Human coders are still better than LLMs

Re:Human coders are still better than LLMs by energy123 on Friday May 30, 2025 @02:00AM (#44133337)

We're seeing diminishing returns in benchmark space, which is partly an artefact of construction, not an absolutely true commentary on how things are progressing.

[ Parent | on HN ]

Re:Human coders are still better than LLMs by klabb3 on Friday May 30, 2025 @12:12PM (#44137651)

Well yes but there is no better way to measure without resorting to pure hearsay. How would you make an accurate assessment of something so inherently vague?

[ Parent | on HN ]

Re:Human coders are still better than LLMs by energy123 on Saturday May 31, 2025 @08:51AM (#44143937)

Alter the benchmark space that we care about, for example focus only on ARC-AGI-2 and then suddenly the gains are no longer diminishing but are accelerating.

[ Parent | on HN ]