Newer models have gotten better at this and it takes longer before they start making things gibberish but all of them have their limit.
And given the size of lots of enterprise codebases like the ones I'm working in, it just is too far away from being useful enough to replace many programmers in my opinion. I'm convinced the CEO's who are saying AI are replacing programmers are just using it as an excuse to downsize while getting investors happy.
It is incredibly powerful for getting things started, but as soon as you have a sketch of a complex system going it loses its grasp on the full picture and do not account for the states outside the small asks you make. This is even more evident when you need to correct it about something or request a change after a large prompt. It just throws all the other stuff out the window and hyperfocus only on that one piece of code that needs changing.
This has been the case since GPT 3, the even their most recent model (forgot the name, the reasoning one) has this issue.
If you're in mostly (or totally) unfamiliar territory, you can end up in a mess, fast.
I was playing around with writing a dead-simple websocket server in go the other evening and it generated some monstrosity with multiple channels (some unused?) and a tangle of goroutines etc.
Quite literally copying the example from Gorilla's source tree and making small changes would have gotten me 90% of the way there, instead I ended up with a mostly opaque pile of code that *looks good* from a distance, but is barely functional.
(This wasn't a serious exercise, I just wanted to see how "far" I could get with Copilot and minimal intervention)