Is anybody else loading their database by tailing log files?
I recently had a brief chat with a couple of developers working on different data-heavy websites, who were both using an interesting pattern for filling their databases.
Their data-gathering components (pulling from external sources like crawlers and APIs) would append new data to the bottom of a log file.
Another process sat doing something like a 'tail -f' on the same file, and parsed and added the updates to the database.
This seems like it might solve some problems for my case:
- Very easy to recreate the database if the schema changes or things blow up, just reread the log files
- Good history for debugging
What worries me is that it feels funky using files for IPC, and I can't find any examples of this being used elsewhere.
So, is anyone else using this pattern, or have any references to it that I'm missing?
(
6 comments
| 5 points
| on HN
)
I'd engineer the crawler to talk to a persistent message queue, and load the database from there. That gives you a lot of flexibility to move loads around, instrument the queue and you're not reinventing things, either.
This would be nasty though if your log files wrapped, the disk they were on ran out of space, etc.
My hazy mental picture is that I'll be creating new local log files regularly and moving older ones to S3 backups, but it feels like there's a lot of edge cases there that I might not think of.
Still feels kinda sketchy...
it's called "the Unix philosophy". invented by Doug McIlroy, probably before you were born