News for hackers. Stuff that matters.
Re:Is anybody else loading their database by tailing log files? by gstar on Thursday September 24, 2009 @06:55PM (#842420)
It's certainly an interesting approach! Depending on the volume, it could have the benefits you described - but it may get a bit old when the volumes increase.

I'd engineer the crawler to talk to a persistent message queue, and load the database from there. That gives you a lot of flexibility to move loads around, instrument the queue and you're not reinventing things, either.

There are no comments on this comment yet.