Salesforce study finds LLM agents flunk CRM and confidentiality tests

Re:Salesforce study finds LLM agents flunk CRM and confident... by handfuloflight on Monday June 16, 2025 @01:04PM (#44291335)

It is an argument: a flawed agent lead to flawed results. A flawed agent does not speak for all agents.

[ Parent | on HN ]

Re:Salesforce study finds LLM agents flunk CRM and confident... by contagiousflow on Monday June 16, 2025 @01:56PM (#44291780)

But the argument should be showing an agent that does in fact pass these tests. You can't just assert that "this one failed, but surely there must be some agent that is perfect, therefore you can't generalize".

[ Parent | on HN ]

Re:Salesforce study finds LLM agents flunk CRM and confident... by handfuloflight on Monday June 16, 2025 @02:51PM (#44292346)

That's not my argument. My argument isn't "surely there must be some agent that is perfect", my argument is this test study can't speak for all agents.

[ Parent | on HN ]

Re:Salesforce study finds LLM agents flunk CRM and confident... by nitwit005 on Monday June 16, 2025 @08:13PM (#44294527)

But no test can. They ran an experiment, they got this result. You can run more experiments if you want.

[ Parent | on HN ]

Re:Salesforce study finds LLM agents flunk CRM and confident... by handfuloflight on Monday June 16, 2025 @08:23PM (#44294582)

I didn't say any test could. I'm pointing out the flaw in the commenters in this thread generalizing the findings.

[ Parent | on HN ]

Re:Salesforce study finds LLM agents flunk CRM and confident... by oblio on Monday June 16, 2025 @01:19PM (#44291451)

The "sufficiently smart compiler" debate, 50 years later :-p

[ Parent | on HN ]

Re:Salesforce study finds LLM agents flunk CRM and confident... by handfuloflight on Monday June 16, 2025 @01:29PM (#44291552)

https://en.wikipedia.org/wiki/Faulty_generalization

[ Parent | on HN ]