Claude 3 beats GPT-4 on Aider's code editing benchmark

Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by stavros on Wednesday March 27, 2024 @03:00PM (#39843228)

I don't think it's a failure of the prompting, but of the tool itself. It just isn't good for any higher-level tasks, and those I know how to do anyway. It's good for menial API work, as a sibling comment says, and using it I can avoid reading API docs, but that's about it.

I tried to have it rewrite in Go a small Python script (a hundred or so lines) that I wrote, and it basically refused, saying "here's the start, you write the rest now". As others said, if it went away it'd be a minor inconvenience of me having to read a bit more documentation.

[ Parent | on HN ]

Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by dontreact on Thursday March 28, 2024 @10:23PM (#39859945)

Yes to me that’s obviously not a good use, and I have experience to know not to attempt hundred line rewrites of files.

I order to reflect on my mental about this. It’s that i more often break down parts problem into sub routines that are 30 or so lines. These are nailed about 95% of the time and when there’s something wrong it’s obvious.

[ Parent | on HN ]

Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by dontreact on Thursday March 28, 2024 @10:24PM (#39859947)

I also wonder if it’s a language effect and the models are just way better at python

[ Parent | on HN ]