I feel like of all the things in this thread, this one is on them. It absolutely is something that LLMs are good at. They have the sample size, examples and docs, all the things. LLMs are particularly good at speaking "their" language, the most surprising thing is that they can do so much more beyond that next token reckoning.

So, yeah, I'm a bit surprised that a shop like Shopify is so sloppy, but absolutely I think they should be able to provide you an LLM to answer your questions. Particularly given some of the Shopify alumni I've interviewed.

That said, some marketing person might have just oversold the capabilities of an LLM that answers most of their core customer questions, rather than one that knows much at all about their API integrations.

Maybe it's right 99% of the time and works well for many of their developers. But this is exactly the point. I just can't trust a system that sometimes gives me the wrong answer.