Genuinely curious what others have experienced but specifically those that are using LLMs for business workflows. It is not to say any system is perfect but for purpose driven data pipelines LLMs can be pretty great.
I've had pretty good success using LLMs for coding and in some ways they are perfect for that. False positives are usually obvious and false negatives don't matter because as long as the LLM finds a solution, it's not a huge deal if there was a better way to do it. Even when the LLM cannot solve the problem at all, it usually produces some useful artifacts for the human to build on.
It might not matter short term, but midterm such debt becomes a huge burden.
I'm sure in 95% of cases it gets it right, but it didn't this time, and I'm not sure how to actually work around that fact.
It probably did save me some time, so I'd call it a mild success; but it didn't save a lot of time, and I only succeeded in the end because I know Salesforce pretty well and was just inexperienced at this one area, so I was able to see where it was probably going off the rails. Someone new to Salesforce would have been hopelessly lost by its advice.
It's understandable that an LLM wouldn't be very good at Salesforce, because there's a lot of bad information in the support forums out there, and the ways of doing things in it have changed multiple times over the years. But that's true of a lot of systems, so it's not an excuse, just a symptom of using LLMs that's probably not going to change.
I find using an agent allows me to "bounce off walls" faster, which is great for software development. The real economic question is how many more situations work the same way.
If I as a human deploy code, it is not certain that it necessarily works - just like with LLMs. The extent is different however.
Cool thing is that, since LLMs are comparatively cheap, I can afford to run the same process a few times, to get a sense of confidence of the response.
In our latest project, the client expressed that our AI aided process was 11 times faster, and much more accurate than their previous process.
Management types seem baffled by the idea we would want this, even if they come around the next hour and say "hey user did something can you tell me what happened".
Like guies ... it's not 100%...
If you cannot tolerate false negatives I don't see how you get around the inaccuracy of LLMs. As long as you can spot false positives and their rate is sufficiently low they are merely an annoyance.
I think this is a good consideration before starting a project leveraging LLMs