>If you're going off of "feel" instead of statistics then you're doing it wrong. Period.

I disagree, at least to a point. Take an experienced Operations/monitoring guy who's been around the block more than once, then sit him in front of the monitoring utilities for a new company developing a new service.

Then, take a total newbie and put him in the same place.

Train them both to equal skill on your tools and operations.

Who do you think will make the most proper calls? Why?

At this point, those statistics and that documentation do not exist yet. What constitutes a "false positive" vs a "drop everything and spin up more VMs and get on the load balancer" can be more of an art than a science, especially when you're first starting out.

As hokey as this sounds, certain systems have a "personality" that varies between installations and companies, that nothing short of day to day use will educate one in.

For operations, I agree. A lot of the numbers you have don't have a rigorous statistical interpretation - for instance is a load average of 20 fine or a problem? Depends if you're looking at the Oracle database.

But the original article was talking about A/B testing, and that is the context that I was thinking about. There you both can and should use statistics.

It's possible (though deceptively difficult) to train one's statistical intuitions to be pretty accurate, or at least accurate enough to give them a positive EV. I was a professional poker player for several years, and this is what becoming a strong player is all about.

There's a lot of intuition and experience that is required to be a talented statistician; once the test has begun things are pretty much locked down (ie, if you know the math there is only one "right" answer from the data). The intuition and experience side of things comes with how the test itself is setup. Are things properly controlled, are you collecting the right data for what you're testing, is the data distributed properly for the test statistic you're using...the list goes on and on and on and on and on.

There are plenty of places that require intuition and these are the places that errors are often introduced. Statistics is an art.

You can build a mathematical intuition. This is a valid thing. You just know how to dig down and prove you are right if asked, so it's ok to leave out details.

Of course you are, my comment was that when you are presented with a decision and the data that was collected and analyzed that went into it, if you've seen a few of these you are much better at catching problems.

If you are the one doing the collecting and analyzing and you just think it feels like enough data but you can't actually reason to that point, then yes, you are doing it wrong.

I always thought statistics existed to give you the "feel". Statistics of more statistics is nonsense!