I'm a senior who generalizes in full stack web development due to my wearing many hats at my company.

ChatGPT and GitHub CoPilot have regularly barfed out garbage for me while I use them. I spend more time double checking whether it's right than I do writing code with it.

It's terrible for my use case where I am doing somewhat complex fixes/refactors everywhere. It just regularly makes up stuff, or gets the wrong API version, or just outright poops out non-existent syntax crap.

This is my experience. These productivity gain numbers are being pulled out of thin air, and certainly not broadly applicable.

> If you're more of a generalist though? I can see productivity gains of 25% maybe.

I really doubt this. Copilot/ChatGPT are not useful at all for the hard problems and they're not trustworthy enough for the small problems. Even for the simplest things I have to comb over the generated code very carefully because I've seen them be subtly wrong dozens of times in my relatively short use. If I have to go over everything they generate with a fine tooth comb it's just easier and less error prone to write it entirely myself.

Mirrors my experience exactly and in CoPilot's case with VSCode until very recently it would suppress normal IntelliSense breaking that whole muscle memory flow.

I'm convinced I've spent more time scrutinizing, undoing, and debugging the CoPilot code than it saved me on typing.

if you're developing code for a large codebase with stable requirements in a company with good engineering practices, and your work output is the working code itself, it's hard to see that it would be very useful.

if you have to glue together multiple unfamiliar APIs and packages, maybe even seeing a couple new ones per day, you often throw away your code, and your work output is something else with writing code only a means to that end, it hugely speeds things up even accounting for debugging the frequent cases where it's subtly wrong.

this latter situation is more common than many software engineers realize.

> if you have to glue together multiple unfamiliar APIs and packages, maybe even seeing a couple new ones per day, you often throw away your code,...

Maybe take the time to be familiar with them? I think that is a description of an unhealthy job. It's like asking an average mechanic to handle optimizing for an f1 car in day 1, then asking him to improve the endurance of a rally car the next day! Then telling them the main point is to market the sponsors, so no need to be perfect.

i don't think the car analogy is good because cars are expensive to build and maintain, usually are used more than once, and it's very bad if they crash. these properties are true of some software but not the kind of software i'm talking about.

maybe a better analogy would be if you travel to new cities a lot, and have to navigate, you don't want to memorize a map of all the city streets and learn in detail the transit schedules on each rail and bus line, you just care about a few routes a few times. and Google Maps can make your life a lot easier. on the other hand if you lived in that city, it would quickly become pointless to take out your phone all the time to plan a commute to work.