Re:Real-Time Log Collection with Fluentd and MongoDB
by kzk_mover on Monday December 05, 2011 @12:38AM
(#3313299)
fluentd's greatest advantage is, it's written in Ruby. So, it's really easy to write the plugin for any datastores.
This is HDFS (Hoop, the HDFS REST gateway) plugin.
* https://github.com/tagomoris/fluent-plugin-hoop
And also, Cassandra plugin are now in under development.
* https://github.com/tomitakazutaka/fluent-plugin-cassandra
You can see the user contributed plugin list here.
MongoDB is very popular, but all the (limited) criticisms of it seem to related to insert performance once the dataset it too big to fit in RAM.
Normally the easy-of-development arguments make up for that, but log files is one of those areas that has a tendency to expand quickly beyond any expectations.
There is a reason why most companies are using HDFS and/or Cassandra for structured log file storage.