Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on AutoDev: Automated AI-driven development by Microsoft. Parent comment
> how this performs against the same benchmark Devin was using
> ...
> Claude 3 Opus already scored around 85-86% on these benchmarks
Devin used SWE-bench, not HumanEval, which kinda implies you said Opus got 85% on SWE-bench which is not true. This was my confusion..
There are no comments on this comment yet.
> how this performs against the same benchmark Devin was using
> ...
> Claude 3 Opus already scored around 85-86% on these benchmarks
Devin used SWE-bench, not HumanEval, which kinda implies you said Opus got 85% on SWE-bench which is not true. This was my confusion..