> We know they cannot do precise things like coding software

Do we? I've found Github Copilot fairly useful, several chatbots have built-in code interpreters to allow the LLM to answer queries by writing and evaluating code, and I've seen demos/startups using AI to generate websites and small basic games.

Certainly there's still a ways to go before reaching human level, but I wouldn't feel safe saying we know it's something LLMs "cannot do".

I agree that they can do things that don’t require too much logic - websites being a prime example, where much of the challenge is in the design and layout and where the response to clicking a button is fairly standard. But if you want to write really complex software like Monte Carl simulations involving highly customized financial products, then you have no choice but to tell it EXACTLY what you want. Free form language requests are never going to get you there. Also, ChTGPT can no doubt find plenty of code snippets dealing with the pressing of a button but there will be precious little available for complex requests. And if the code starts to stretch into tens of thousands of lines, then someone needs to look it over.

I'd agree that non-shallow reasoning will probably be the biggest challenge to overcome.

> you have no choice but to tell it EXACTLY what you want. Free form language requests are never going to get you there

I don't see any fundamental limitation in this regard compared to the level of detail you'd need to give a human programmer. With existing integrations you can already ask along the lines of "fit a model to this dataset [attachment]" and have it infer a lot of the details, then iterate on it with clarification as you could a human programmer if needed.