News for hackers. Stuff that matters.
Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by thorum on Thursday May 15, 2025 @05:08PM (#43999358)
GPT 4.1 and 4o score very low on the Aider coding benchmark. You only start to get acceptable results with models that score 70%+ in my experience. Even then, don't expect it to do anything complex without a lot of hand-holding. You start to get a sense for what works and what doesn't.

https://aider.chat/docs/leaderboards/

Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by bjt12345 on Thursday May 15, 2025 @08:42PM (#44000765)
That been said, Claude Sonnet 3.7 seems to do very well at a recursive approach to writing a program whereas other models don't fare as well.
Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by k__ on Friday May 16, 2025 @05:54AM (#44003517)
Sonnet 3.7 was SOTA for quite some time. I built some nice charts with it. It's a rather simple task, but quite LoC-intensive.