Well, of course. Realistically, I would not expect AI systems like this to be very useful for novel cutting-edge scientific results, proving mathematical theorems etc. in the next few years.

But this is not the majority of what software developers are doing and working on today. Most have a set of features or goals to implement using code satisfying certain constraints, which is what current reasoning AI models seem to be able to do very well. Of course, this test was not rigorous in any meaningful way, but it really changed my mind on the pace of this technology.