We can't measure small changes and we aren't great at comparing across orgs.
However, at the director level we can certainly see a 50% or 100% productivity improvement in our teams and with individuals in our teams.
We aren't seeing changes of this magnitude because they don't exist.
Perhaps developers are now slacking off.
Perhaps we have added more meetings because developers have more free time.
Or perhaps developers were never the bottleneck.
We can see large productivity improvements when we make simple changes like having product managers join the developers daily standup meetings. We can even measure productivity improvements from Slacks/Zooms auto-summary features. Yet gpt/copilot doesn't even register.
While not code generation, this auto-summary is powered by the same tech. I think using it to sift through and surface relevant information, as opposed to generation of new things, will have the biggest impact.
By far the greatest value I get out of LLMs is asking them to help me understand code written by others. I feel like this is an under-appreciated use. How long has this feature been in Copilot? Since February or so? Are people using it? I do not use Copilot.
Now that's dangerous thinking, but I think you are onto something.
Edit: and now I see chillfox made the same point.
You can't measure productivity for shit, otherwise companies would look entirely differently. Starting from me not having to do my own finances or event planning or hundred other things that are not my job description, not my specialty, and which were done by dedicated staff just a few decades ago, before tech "improved office productivity".
> We aren't seeing the developers who start using copilot/gpt rush ahead of their peers.
That's because individual productivity is usually constrained by team productivity. Devs rushing ahead of their teammates makes the team dysfunctional.
> We aren't seeing any ability to cut back on developer spend.
Devs aren't stupid. They're not going to give you an opportunity if they can avoid it.
> We aren't seeing anything positive yet and many developers have been using copilot/gpt for >1 year.
My belief is that's because you aren't measuring the right things. But then, no one is. This is a problem well-known to be unsolved.