News for hackers. Stuff that matters.
Re:Salesforce study finds LLM agents flunk CRM and confident... by skybrian on Monday June 16, 2025 @11:44AM (#44290680)
Publishing new benchmarks seems useful? If LLM’s improve on this benchmark (and they probably will, like they have on many others) then they’ll need less work on prompting, etc.
Re:Salesforce study finds LLM agents flunk CRM and confident... by CityOfThrowaway on Monday June 16, 2025 @12:40PM (#44291142)
The benchmark is useful, but the conclusion of the write-up is that current generation LLMs can't solve the problem. That's not a valid conclusion to draw. The results here tell us mostly about the skill of the agent-designer, not the capabilities of the model.