Re:Real-Time Log Collection with Fluentd and MongoDB
by alexchamberlain on Monday December 05, 2011 @03:05AM
(#3313673)
Have you got more details about overcoming the global write lock?
Re:Real-Time Log Collection with Fluentd and MongoDB
by ngokevin on Monday December 05, 2011 @03:08AM
(#3313682)
Oh sorry, when I mean overcome the global write lock, I don't mean getting rid of it, but simply allowing me to querying a replicated slave database while the master database is getting hundreds of writes a second...so the writes don't block the reads.
github.com/ngokevin/netshed
It is written in Python current parses out fields from several types of logs (such as dhcpd). It is initially set up to read from named pipes (it has a tail function as well). Each type of log is dumped to its own database, and each date has its own collection. I have it set up with a master/slave configuration to overcome the global write lock. It has functions to simulate capped collections by days. It is followed with a Django frontend for querying via PyMongo.
This version is several weeks old and I will push out a new one soon.