I needed to do something like this and I ended up settling on a custom workflow connected through zeromq dumping data into into sharded sqlite databases. Single process handled writing to each database and lots of reads were mapped out and then reduced (also via zmq). This handled around 14 billion rows when I left that job. If you want to talk some more about it, my gmail is the same as my username.

Thanks - I have emailed you.

Have you looked at hyperdex?

Yes, it's the closest to my current needs. But it's unlikely I will be able to use it currently as it's still pretty nascent and I don't feel comfortable just yet. Anyhow - I am looking to run some benchmarks on it next week.