Re:Ask HN: What tools do you use to monitor your LAMP server...
by z2amiller on Sunday October 24, 2010 @01:34AM
(#1825324)
One thing I've done in the past to help prevent 'configuration rot' of nagios configs is to hook it into the same files that drove our deployments - when the deployment changes, the nagios config changes with it. Nagios supports template-based configs so once the tests are developed it isn't too hard to write some automation that spits out a config file. This worked pretty well for a site with ~350 servers and devices (switches, APC rack PDU's) with ~3000 tests.
If you don't have a full time sys admin it's also one of those things that is very easy to put off or delay or just not keep on top of.
As for the data loss I can't think of a time I've ever wanted to see resource statistics for over a year ago and down times and outages are kept in a problem tracking system.