It is simply unable to do anything novel. I've had arguments with friends about this, specifically in reference to the paper "Sparks of Artificial General Intelligence: Early experiments with GPT-4", which is wonderful and presents some amazing capabilities, many of which I use constantly for work every single day. But, these capabilities seem to all be within the range of data that it's trained on. Or they can be seen as interpolations, which are as novel as the prompter can suggest, but which are clearly derivatives of modes in the data and not of deep understanding of abstract concepts.

It's amazing stuff. But it totally fails to take the prompter anywhere new without extensive support, and it is still at a very shallow level of understanding with complex topics that require precision. For instance, turning a mathematical description of a completely novel (or just rare or unusual) algorithm into code will almost never work, and is more likely to generate a mess that takes lots of effort to clean up. And it's also extremely hard to get the model to self reflect and stop when it doesn't understand something. It is at present almost incapable of saying "I don't have enough information or structure to do X".

If we are already as deep into a realm of diminishing marginal returns as the GPT-4 white paper suggests, we might indeed be approaching a limit for this specific approach. No wonder someone is trying to dig a regulatory moat as fast as they can!