But the devs here are all very experienced senior-level engineers. Perhaps that's why we aren't seeing any gains?
I think many software developers have this romantic, idealized notion that their entire job should be furiously typing code into a computer, and that anything else they have to grudgingly do is unproductive overhead.
My experience was that it was about as useful as searching Google and Stack Overflow without knowing what a good answer looks like. You copy and paste a lot of code, a lot of it works, but inevitably, something breaks and you have no idea why. So you still have to go back to "the hard way" of reading the documentation and building an understanding.
I'm sure things will improve over time, especially for products built specifically for code generation, but my first and second impression is that so far, LLMs don't actually lower the bar for junior programmers.
Otherwise you're suggesting that these senior devs are using a tool which provides no value whatsoever which seems unlikely.
I'd say it depends on the task more than on dev level. If it's maintaining, improving, or refactoring of a large project LLM can't help much. When it comes to small utilities GPT-4 is a time saver. You can even write a small playable game with it in a matter of hours. Using any popular language. Simple web pages, JS are much easier. With ChatGPT-4 I've learned how to do certain things using ffmpeg. Without it would take hours of googling.
PS: talking about ChatGPT degradation with the time. I didn't see it on programming tasks. But it became noticeably faster.
So if anyone gains from this, is the companies paying under-market, and therefore typically getting somewhat substandard devs to do boring things. There's major savings in productivity there, precisely because the training for those tasks is so good. If you had your highest performance developers spending a significant amount of time on that kind of busywork, you weren't getting that great of a performance in the first place.