News for hackers. Stuff that matters.

A comment on Claude 3 beats GPT-4 on Aider's code editing benchmark. Parent comment

Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by CSMastermind on Wednesday March 27, 2024 @09:19AM (#39838657)
Is it a clear improvement?

I run every query through Claude 3, GPT-4, and Gemini Advanced just to compare results.

Claude 3 and GPT-4 seem roughly on par with each other while Gemini is very clearly inferior.

I've run 47 queries in the last month. I marked Claude as doing better than GPT-4 on 2 of those and worse on 3 with the rest being roughly equal.

I wouldn't say it's a clear improvement so much as its an on par competitor.

[ Parent | on HN ]
Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by whiplash451 on Wednesday March 27, 2024 @09:26AM (#39838754)
Just curious: do you also benchmark Mistral?
[ Parent | on HN ]
Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by CSMastermind on Wednesday March 27, 2024 @10:19AM (#39839498)
No though if there's an easy way for me to do it I'd add it to the list. Is someone offering a web interface?
[ Parent | on HN ]
Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by stavros on Wednesday March 27, 2024 @09:21AM (#39838684)
Which Claude 3 version is this?
[ Parent | on HN ]
Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by nycdatasci on Wednesday March 27, 2024 @10:26AM (#39839594)
Opus.
[ Parent | on HN ]
Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by CSMastermind on Wednesday March 27, 2024 @10:19AM (#39839488)
Opus
[ Parent | on HN ]