News for hackers. Stuff that matters.

A comment on Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. Parent comment

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by usaar333 on Tuesday October 22, 2024 @09:16PM (#41920623)
FWIW, the refactor benchmark is quite mechanical - it just stresses reliability of LLMs over long context windows:

Questions are variants of:

Refactor the _set_csrf_cookie method in the CsrfViewMiddleware class to be a stand alone, top level function. Name the new function _set_csrf_cookie, exactly the same name as the existing method. Update any existing self._set_csrf_cookie calls to work with the new _set_csrf_cookie function.

[ Parent | on HN ]
Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by jstummbillig on Wednesday October 23, 2024 @02:10AM (#41922296)
Assuming that that is indeed what most of the benchmark does: If the LLMs are as bad as it as the numbers suggest, then it seems like a perfectly good benchmark. I would definitely want them to be able to do stuff like that when I let them write my code.
[ Parent | on HN ]