News for hackers. Stuff that matters.
Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by dontreact on Wednesday March 27, 2024 @10:46AM (#39839921)
It’s a tool you learn to use and improve with. How many hours of practice have you given the tool?

Personally I’ve been practicing for almost 3 years now going back to CoPilot so I would guess I have at very minimum a few hundred hours of thinking about how (what to prompt for, what to check the output for) and probably more importantly when (how to decompose the task I am doing into parts for which some the LLM can reliably execute) to use the tool.

Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by stavros on Wednesday March 27, 2024 @03:00PM (#39843228)
I don't think it's a failure of the prompting, but of the tool itself. It just isn't good for any higher-level tasks, and those I know how to do anyway. It's good for menial API work, as a sibling comment says, and using it I can avoid reading API docs, but that's about it.

I tried to have it rewrite in Go a small Python script (a hundred or so lines) that I wrote, and it basically refused, saying "here's the start, you write the rest now". As others said, if it went away it'd be a minor inconvenience of me having to read a bit more documentation.

Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by dontreact on Thursday March 28, 2024 @10:23PM (#39859945)
Yes to me that’s obviously not a good use, and I have experience to know not to attempt hundred line rewrites of files.

I order to reflect on my mental about this. It’s that i more often break down parts problem into sub routines that are 30 or so lines. These are nailed about 95% of the time and when there’s something wrong it’s obvious.

Re:Claude 3 beats GPT-4 on Aider's code editing benchmark by dontreact on Thursday March 28, 2024 @10:24PM (#39859947)
I also wonder if it’s a language effect and the models are just way better at python