I think the phrase you are looking for is that LLMs/GPTs are "a solution in search of a problem". They suffer from the exact same that every neural network has in that their behaviour is unpredictable in corner cases. That's not something one can trust their business to. There have been enough legal issues created by simple chat bots agreeing to deals with customers that I don't see how one can build trustworthy automation on top of LLMs/GPTs without a human in the loop to catch the failures. Of course, there will be plenty of people that will do that regardless.

100% agree. You can even see it here on HN. When was the last time you saw a post where someone was excited about something they were actually _doing_ with and LLM? Mostly I see threads about business hype and the periodic conversation started by a confused person saying “so uh I tried LLMs and how do you actually get any value out of them?” And I think programming is actually the domain that LLMs will be MOST effective in.