News for hackers. Stuff that matters.
Re:Real-Time Log Collection with Fluentd and MongoDB by ngokevin on Monday December 05, 2011 @02:35AM (#3313606)
I have a syslog-ng -> MongoDB project that I've been working on at my university.

github.com/ngokevin/netshed

It is written in Python current parses out fields from several types of logs (such as dhcpd). It is initially set up to read from named pipes (it has a tail function as well). Each type of log is dumped to its own database, and each date has its own collection. I have it set up with a master/slave configuration to overcome the global write lock. It has functions to simulate capped collections by days. It is followed with a Django frontend for querying via PyMongo.

This version is several weeks old and I will push out a new one soon.

Re:Real-Time Log Collection with Fluentd and MongoDB by alexchamberlain on Monday December 05, 2011 @03:05AM (#3313673)
Have you got more details about overcoming the global write lock?
Re:Real-Time Log Collection with Fluentd and MongoDB by ngokevin on Monday December 05, 2011 @03:08AM (#3313682)
Oh sorry, when I mean overcome the global write lock, I don't mean getting rid of it, but simply allowing me to querying a replicated slave database while the master database is getting hundreds of writes a second...so the writes don't block the reads.