What bothers me with this kind Of culture is that it quickly becomes a gatekeeper for ideas you don’t like. Like an idea? Push for it without data. Don’t like an idea? Ask for more data.

This is what the author is saying whether they realize it or not. This is the advice for dealing with a ZEBRA. Data as a cudgel.

Excellent point and potentially good material for a separate post - I totally agree. On that note almost any kind of culture can be (mis)used to create gatekeepers.

Agree, it's just mumbo jumbo to most people. Being really data driven in many realms falls into the category of Complicated or Chaos (in terms of the Cynefin framework) so often times being truly scientifically data driven is a matter of a PhD thesis. The only true recourse many companies have is intuition and trial and error which according to Cynefin is the most effective means of operating in the chaos realm

Genuine question – what's the alternative to trying? Being data-driven is definitely hard but imho teams should be commended for striving towards it.

Even imperfect shifts towards data-driven decision making can be valuable as well (eg maybe you have great data for optimizing your onboarding flow, even if knowledge of which features most improve retention or database performance are still black boxes).

I think part of the issue is that rampant innumeracy makes half-assed attempts at being data-driven actually worse than nothing. The reason is that without clean data we fall back to the old standard: gut feel + pet theories. Which of course is often not great, but then again almost always has more than zero correlation with reality than random. Poorly executed data collection and analysis on the other hand, I think drives many teams at full speed right off a cliff. All the randomness of flying blind with the confidence of feeling certain.

Being data driven can be as simple as sales telling you how many people are asking for a feature as opposed to "everybody".

Even in that simple of an example, we lack the data we need for that number to matter much, and the fact that it's a concrete statistic can easily lead to prioritizing one feature or another using that imperfect data as a cudgel.

In particular, you'd probably want to know how a feature would impact future sales, survivorship curves for current customers, and survivorship curves for whichever kind of customer would sign on with the new feature (treating a single feature in isolation for simplicity). This is especially important when comparing multiple features to put on the roadmap because it's easy for one idea to be simple to imagine and better than the status quo (hence asked for by many customers) but be nowhere near an optimal solution to whichever problem is being solved.

Having that kind of customer insight is better than not having it, as long as such data is used appropriately -- without additional data it can't do much more that guide or refine gut feelings and insights, and attempting to do otherwise is a recipe for an inferior product.

Great point, and the post never intended to talk about this topic. I think there's great merit to 'gut feeling' from field experts, even if they cannot support it with data at the moment, maybe because the nature of the thing is exploratory at that certain stage or it's simply too chaotic. Experience and intuition coming from experts tenured in relevant areas is priceless - but it doesn't have to come with hierarchy or power based authority. When people naturally respect someone they will listen. No need to force it.

I clearly failed with the post (by not elaborating) on this point - I never meant forcing fake data where it's not possible. I'd still rather have this conversation:

A: We should definitely do X!

B: Why? Why not Y?

A: Because I'm the authority in the room

B: Well we talked to sales and customers and Y clearly has Z impact, on the other hand we saw no evidence of customers needing X.

than

A: We should definitely do X!

B: Why? Why not Y?

A: Because I'm the authority in the room

B: Whelp, ok I guess

my xp at big software corps has been that technical decisions are often heavily data driven. product decisions are rarely so.

i think about how there will deep dive root cause analysis for technical misses, but there is nothing like that for product misses.

There should be analyses of product misses in a well functioning product org, though getting to a concrete “root cause” can be a lot harder and may not need to be the goal.

You should at least be trying to understand, to the extent possible, why something didn’t work and then using that to better understand the problem you’re were originally trying to solve.

Data-driven works well for cheap services with many customers. Try it with expensive services with few customers and your experiment quickly becomes statistically worthless and expensive.

Also, data-driven does never work for a product. If you sell a product, your customers do not want to be monitored. They also do not want to randomly get an inferior version of the very product they paid for just to satisfy your research goal. So your only realistic test case would be to offer a second product and see how it sells. Or to pay for expensive surveys.

>Most A/B testing is blindness dressed in nice clothes.

... and harassing you.

Excellent definition.

A/B testing is but one way of gathering data. Also, it usually comes after the decision to actually invest in building _something_ out (even if it's only a prototype/MVP). There are other types of data such as sizing the opportunity based on market research, competitor analysis or simply asking your customers - among many other. This shouldn't be trusted blindly either, sure. Never said that.