I don't think you're disagreeing with the article. He shows a real-time dashboard of ops stats, and then says:
> It's important to divorce the concepts of operational metrics and product analytics.
Real-time ops statistics help solve ops crises, but product analytics can afford a little more consideration.
I don't think operational and product analytics are different in principle. In both cases, the best speed for analytics matches the speed of taking action on the results.
In ops, there are people whose very job is to respond to a server crash in 2 minutes. In product design, there are no actions taken within minutes of seeing the change in data - so faster analytics just wastes time on refresh-procrastination, and encourages needlessly hasty decisions.
If you're going off of "feel" instead of statistics then you're doing it wrong. Period.
I disagree, at least to a point. Take an experienced Operations/monitoring guy who's been around the block more than once, then sit him in front of the monitoring utilities for a new company developing a new service.
Then, take a total newbie and put him in the same place.
Train them both to equal skill on your tools and operations.
Who do you think will make the most proper calls? Why?
At this point, those statistics and that documentation do not exist yet. What constitutes a "false positive" vs a "drop everything and spin up more VMs and get on the load balancer" can be more of an art than a science, especially when you're first starting out.
As hokey as this sounds, certain systems have a "personality" that varies between installations and companies, that nothing short of day to day use will educate one in.
But the original article was talking about A/B testing, and that is the context that I was thinking about. There you both can and should use statistics.
There are plenty of places that require intuition and these are the places that errors are often introduced. Statistics is an art.
If you are the one doing the collecting and analyzing and you just think it feels like enough data but you can't actually reason to that point, then yes, you are doing it wrong.
The central theme is a good one though, tactics or strategies have an innate timeline associated with them, and deciding on tactics or strategies with data that doesn't have a similar timeline leads to poor decisions. The coin flip example in the article is a great one.
Ideally one could say "What is the shortest interval of coin flips I can measure to 'accurately' determine a fair coin?" And realize that accuracy asymptotically approaches 100%. One of the things that separate experienced people from inexperienced ones are having lived through a number of these 'collect-analyze-decide' cycles and getting a feel for how much data is enough.