In science, measuring things until you find a benefit is called p-hacking. Every extra test you do that splits your data along a different dimension, is another independent opportunity for "random chance" to look like positive signal.

There is no programming project in existence with enough developers working on it, that developer-productivity data derived from a change to it would not be considered "underpowered" for the sake of proving anything.

The obsession with measuring is hilarious. There are plenty of things in life (and jobs) that aren't measurable and are worth doing. Probably all of the important things are actually unmeasurable. Think about it this way, if its so easy you can measure it, it probably isn't very important in the grand scheme of things.

No metric can escape gaming when you apply it to rational actors (Campbell's Law / Goodhart's Law). Blind devotion to metrics is just as bad as no metrics at all.

I was just yesterday discussing the opportunity cost of infrastructure changes, as a new team member was bemoaning our out of date patterns...

A high impact infra change will often inconvenience dozens of people and distract from feature work... You know, the shit people actually care about... (this is analogous to how "Twitter, but written in Golang" appeals to approximately no one.)

> find another way to measure developer productivity

And solve the halting problem while you're at it