News for hackers. Stuff that matters.
Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by simonw on Thursday May 15, 2025 @05:13PM (#43999401)
"It used a deprecated package"

That's because models have training cut-off dates. It's important to take those into account when working with them: https://simonwillison.net/2025/Mar/11/using-llms-for-code/#a...

I've switched to o4-mini-high via ChatGPT as my default model for a lot of code because it can use its search function to lookup the latest documentation.

You can tell it "look up the most recent version of library X and use that" and it will often work!

I even used it for a frustrating upgrade recently - I pasted in some previous code and prompted this:

This code needs to be upgraded to the new recommended JavaScript library from Google. Figure out what that is and then look up enough documentation to port this code to it.

It did exactly what I asked: https://simonwillison.net/2025/Apr/21/ai-assisted-search/#la...

Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by kgeist on Thursday May 15, 2025 @05:27PM (#43999494)
>That's because models have training cut-off dates

When I pointed out that it used a deprecated package, it agreed and even cited the correct version after which it was deprecated (way back in 2021). So it knows it's deprecated, but the next-token prediction (without reasoning or tools) still can't connect the dots when much of the training data (before 2021) uses that package as if it's still acceptable.

>I've switched to o4-mini-high via ChatGPT as my default model for a lot of code because it can use its search function to lookup the latest documentation.

Thanks for the tip!

Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by jmcpheron on Thursday May 15, 2025 @05:53PM (#43999708)
>I've switched to o4-mini-high via ChatGPT as my default model for a lot of code because it can use its search function to lookup the latest documentation.

That is such a useful distinction. I like to think I'm keeping up with this stuff, but the '4o' versus 'o4' still throws me.

Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by tptacek on Thursday May 15, 2025 @11:17PM (#44001517)
Model naming is absolutely maddening.
Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by fragmede on Thursday May 15, 2025 @05:39PM (#43999578)
There's still skill involved with using the LLM in coding. In this case, o4-mini-high might do the trick, but the easier answer that worry's with other models is to include the high level library documentation yourself as context and it'll use that API.
Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by th0ma5 on Thursday May 15, 2025 @09:38PM (#44001060)
Whate besides anecdote makes you think a different model will be anything but marginally incrementally better?
Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by sagarpatil on Thursday May 15, 2025 @11:17PM (#44001520)
Context7 MCP solves this. Use it with Cursor/Windsurf.
Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by mbesto on Friday May 16, 2025 @09:53AM (#44005551)
> That's because models have training cut-off dates.

Which is precisely the issue with the idea of LLMs completely replacing human engineers. It doesn't understand this context unless a human tells it to understand that context.

Re:The Unreasonable Effectiveness of an LLM Agent Loop with ... by simonw on Saturday May 17, 2025 @09:08PM (#44018209)
Right: the idea that LLMs are a replacement for human engineers is deeply flawed in my opinion.