News for hackers. Stuff that matters.

A comment on AutoDev: Automated AI-driven development by Microsoft. Parent comment

Re:AutoDev: Automated AI-driven development by Microsoft by PodgieTar on Saturday March 16, 2024 @09:40AM (#39725819)
https://paperswithcode.com/sota/code-generation-on-humaneval
[ Parent | on HN ]
Re:AutoDev: Automated AI-driven development by Microsoft by riku_iki on Saturday March 16, 2024 @01:14PM (#39727671)
Human Eval is very different to SWE-Bench on which Devin is tested
[ Parent | on HN ]
Re:AutoDev: Automated AI-driven development by Microsoft by PodgieTar on Sunday March 17, 2024 @05:51AM (#39733168)
I didn't say it was the same, I compared non-agentic Claude to this. This used HumanEval.
[ Parent | on HN ]
Re:AutoDev: Automated AI-driven development by Microsoft by riku_iki on Monday March 18, 2024 @07:05PM (#39751121)
You said:

> how this performs against the same benchmark Devin was using

> ...

> Claude 3 Opus already scored around 85-86% on these benchmarks

Devin used SWE-bench, not HumanEval, which kinda implies you said Opus got 85% on SWE-bench which is not true. This was my confusion..

[ Parent | on HN ]