Obviously, you need metrics to monitor the metrics.
But more seriously, I've attacked this kind of problem in the past by agreeing on a common set of first-look metrics. What is a one page set of data you start with when trying to identify a problem in a given system/subsystem.
After than, other metrics must exist to solve or monitor a specific issue, or be in reusable shared sub-reports with some stated overview. After than, you can think about automatic deletion after a specific timeframe if they're in sandboxes and not connected to some documented useful purpose.
Only way is to make it self documenting, when a metric is set up, when you are navigating the graphs etc there should be some text defining where this data came from. It also depends if lack of data means something to your system. In my system a lack of data for a metric for a time span x implies that something is wrong and should be alerted. In some systems this might not matter. This provides a self fulfilling habit of removing dead metrics
Re:Graph everything with Graphite
by yummyfajitas on Friday December 12, 2014 @06:04AM
(#8739452)
Looks interesting, thanks!
Re:Graph everything with Graphite
by apurvadave on Friday December 12, 2014 @03:28PM
(#8742503)
We are working on building a system that combines events and metrics (www.jut.io), and one of the things we've been thinking about is how to store metadata about your metrics. It's not only stuff like min/max values, but almost anything you could think of - developer, feature name, expected frequency, etc. Would a concept like this be useful?
Now, we have another problem. There are over 2 million metrics in our monitoring system and no one knows what most of them mean. Some graphs have been set up for features that don't exist anymore, other graphs were set up by developers who have already quit, there are lot's of duplicated metrics and in general it is a mess. So we are currently working on this problem. I still would like to mention this is a better problem then not having metrics at all, but still a problem.
If you ever had a similar situation, I will be thankful if you could share your experience on how you solved it.