There's also the flip side - metrics-driven development used to keep other people from thinking too hard and scrutinizing work. At my last company, many product decisions were justified using a litany of metrics including AB tests, which people just swallowed because "the data speaks for itself!"
I once took a look at the app we were using to run the AB tests and not a single one had reached statistical significance. The company was basing its decisions on data considered garbage even by the AB app's naive p-value algorithm. I thought that after dozens of A/B tests at least one would be statistically significant just by dumb luck but no, not a single test had even the veneer of credibility.
There's so much noise in business data that it's like reading tea leaves, except no one can prove it because by the time they gather enough data they realize that the management and/or market dynamics have introduced even more noise so everyone keeps chugging along, blissfully unaware that they're indulging in statistical fantasies and reinforcing whatever beliefs they already hold.
But we killed QA, and Ken Schwaber gave us a treadmill to run on forever where structurally, nobody has time to think past the end of next month. So Product has been running progressively more roughshod over everything for the last 10 years. And since 50% of devs have 5 years or less of experience, that means most people don't know that things used to be better.
A company that is famous for being metrics driven and having teams that optimise for their own metrics is Facebook. You can tell that the product has "seams". Each page and widget feels like it was individually optimized and then stitched together. Overall it somehow feels off.
Apple puts design on top and has a kind of dictatorial veto over all aspects of the product. Hence products made by Apple feel seamless and cohesive. They are also more polished because when you solely rely on metrics you make it as good as it needs to be and then stop.
I feel like Apple’s method is much better than blindly trying to interpret data and let that drive you over the cliff. Data isn’t a replacement for understanding of your product and users.
The more surprising conclusion is that the dumb, data-driven rules outperform experts even when the experts get access to the same data and the decision made by the data-driven rule. Why? Humans have a tendency to think they're better than dumb rules and override these dumb rules in ways that make their decision ultimately worse.
The result has remained remarkably consistent across many fields where people were sure it wouldn't apply. At this point it's stable enough that I think the conclusion is a good base assumption, and it would take serious evidence to show that it does not apply in any given field.
I would be grateful if you can share Apples general approach to product development.
https://hbr.org/2020/11/how-apple-is-organized-for-innovatio...
After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake. Obviously, metrics can be a useful supplement to a deep understanding of the product, the domain, the users, and the company, but keep it on a leash.