The idea that "When a measure becomes a target, it ceases to be a good measure" is something that every data scientist/statistician knows, but almost none heed in practice.
The worse lead companies that I've worked for are the ones that claim to be "data driven". Countless dashboards showing various progress towards various targets without even a hint of understanding what the big picture might even be.
One of the biggest insights I've had over a career working in data science is that the person solving a problem based on years of experience without any numbers backing their decisions almost always is making choices close enough to optimal that it isn't worth the extra energy to push it optimal.
An example is that the person selling hot dogs at the park is probably pricing them nearly optimal. You could bring in a team of dynamic pricing experts, build a data center to mine costumer data, and I'm willing to bet a few hot dogs that difference between the model optimal price and what the hot dog vendor is selling is not enough to justify the cost of figuring out the difference.
I likewise would not be surprised if the real, long run benefit of A/B testing does not justify the cost of both employee time and especially the SaaS products that help manage these processes... but let's not do that analysis because my salary depends on no one checking this.
I'm not sure I'd agree that people are often operating "close enough to optimal", but I would definitely agree that integrating experimentation is hard enough that sometimes the effort (or mistakes) you can introduce will cause more problems than you're helping. But I think this is more a function of how poor people are at the mechanics and the mindset of running experiments than that they're doing good enough pricing hot dogs. Experiments in many places are looked at for either CYA or boasting about quarterly results and not to truly learn/grow/improve.
It always takes longer than you expect, even when you take into account Hofstadter's Law.
I would still consider an indirect measure or proxy measure of a thing to be "a measure of the thing".
The error bars of our measurements may be large, and the there may be many confounding variables to the measurement, but it is still a measurement.
Having said that, I agree with the overall sentiment above. We often pick metrics that don't measure what we think they do or do it so inaccurately as to be useless. And at the same time we invest too little in finding the right metrics before they become ingrained in the org structure.
You've never had one steak you prefer to another without doing quantitative analysis on it?
I love making cocktails and I can definitely tell better from worse, noting qualitative differences (a bit dry, too sweet, mouthfeel a bit thick) with absolutely no quantitative measurements involved.
I can certainly tell the difference between a good and bad violinist without measuring anything.
From a different perspective: many of my favorite movies have a 3.5 star rating on Amazon, in this case the measure does not correlate well with my sense of improvement.
I can't stand this cliche. You can improve things you can't measure, we do this all the time, and the things worth improving most are often hard to measure. Setting objective goals are important to keep us honest with ourselves, but in that same vein, we should continuously acknowledge that, unless we are working in hard sciences, we are usually measuring proxies to what we really care about, and it's often hard to pick a representative measure.
As an example:
> Measures can be as simple as number of experiments per person per team or something more complicated.
I've seen this exact scenario in a large company I used to work for. The outcome was that some teams would hit their goal by running garbage experiments. It was a net negative for the product, because garbage still occasionally shows statistical significance. Acknowledge that the measure is an imperfect proxy, identify in what ways the measure could fail to represent the true desired outcome, and control for those (in this case, e.g. some oversight on experiment quality).