Ryan Smith's "Building Metrics From Log Data" is an interesting talk about this:
Also check out Heroku's Lumbermill project, which handles extracting router metrics on their platform:
https://github.com/heroku/lumbermill
If you're on AWS, CloudWatch is capable of ingesting log data and extracting metrics via pattern-matching:
http://docs.aws.amazon.com/AmazonCloudWatch/latest/Developer...
When I used Graphite with Nagios, I bypassed all the Nagios data-collection and graphing features. Instead, I funneled all the data into Graphite and used check-graphite to alert on it:
I've used collectd's tail module for this. See https://collectd.org/wiki/index.php/Tail for details.
Has anyone integrated a graphing system like this in to Graylog/Logstash and care to share some lessons or advice on how you did it?
Finally, monitoring suites like Nagios an Zabbix: they have their own data-collection and graphing features. When tools like Graphite are used in conjunction with these, would you bypass them entirely? Or would you leverage their data-collection features to first grabb the data and then somehow funnel that data in to Graphite?
If it's the former, what do you use instaed of all thosebuilt-in data collection tools? If it's the latter, how do you do it?