News for hackers. Stuff that matters.
Re:The Virtues of Monitoring by Uchikoma on Wednesday January 05, 2011 @10:21AM (#2071199)
Monitoring is one of the most important thing.

Rule of thumb: Think of everything you wish you had monitored after a crisis has happened (which might be "everything").

Re:The Virtues of Monitoring by riffraff on Wednesday January 05, 2011 @10:51AM (#2071301)
the problem is you will probably think of monitoring the stuff that you wished you had monitored only _after_ the crisis happened, people are built in a way that makes it hard for us to think of what may go wrong.

So another simple rule I learned with time is to trust/understand the defaults,plugins,knobs,metrics that come with well known monitoring systems ("why the hell should I monitor _that_?"). This way you use the experience of other people as a backup for your own.

Re:The Virtues of Monitoring by ojilles on Wednesday January 05, 2011 @12:43PM (#2071674)
How about starting with the application/business metrics first (as those are presumably easier to articulate). As things fail over time move down the stack (infra/system) to get earlier warnings?
Re:The Virtues of Monitoring by Uchikoma on Wednesday January 05, 2011 @11:45AM (#2071469)
What I have learned: Take Munin (or your solution of choice) and install all plugins for infrastructure to use. It's hard to monitor too much, only too little.