As a result I decided to start using those LLMs myself, to see what they could do and how quickly they would make me obsolete. Two months later, I feel a lot less worried. Yeah they're really good for some things and they do save me a lot of time in some cases, but I wouldn't trust them to develop a complex application from the ground up without oversight. If it's a one-off prototype thing then maybe, but for a serious long-lived project it's going to be utter chaos and confusion.
One of the reasons the bots are so good at pure coding is that code is designed to be machine-friendly by definition, and the feedback loops are extremely short and cheap. Oh you forgot to add an #include? Read the compiler's error message and fix it. Takes 2s. I suspect that's also true for the Mathematical proof work to some extent, your Lean either checks or it doesn't. As soon as problems become more organic and less immediately testable, you can see models losing their footing. When there isn't a clear test to decide between A and B, the models easily panic, loop, overthink, underthink, miss alternatives etc...
Of course it may change. Maybe next year Opus 15.5 will truly outrank me. But there's still a long way to go IMO.
SWE's have to guide the AI, look at all those AI PR's, many of them make very bad assumptions.