Re:Whom the Gods Would Destroy, They First Give Real-time An...
by btilly on Wednesday January 09, 2013 @06:35PM
(#5034260)
For operations, I agree. A lot of the numbers you have don't have a rigorous statistical interpretation - for instance is a load average of 20 fine or a problem? Depends if you're looking at the Oracle database.
But the original article was talking about A/B testing, and that is the context that I was thinking about. There you both can and should use statistics.
I disagree, at least to a point. Take an experienced Operations/monitoring guy who's been around the block more than once, then sit him in front of the monitoring utilities for a new company developing a new service.
Then, take a total newbie and put him in the same place.
Train them both to equal skill on your tools and operations.
Who do you think will make the most proper calls? Why?
At this point, those statistics and that documentation do not exist yet. What constitutes a "false positive" vs a "drop everything and spin up more VMs and get on the load balancer" can be more of an art than a science, especially when you're first starting out.
As hokey as this sounds, certain systems have a "personality" that varies between installations and companies, that nothing short of day to day use will educate one in.