News for hackers. Stuff that matters.
Re:Analyze terabytes of data with just a click of a button by emidln on Friday August 17, 2012 @03:10PM (#4398195)
I needed to do something like this and I ended up settling on a custom workflow connected through zeromq dumping data into into sharded sqlite databases. Single process handled writing to each database and lots of reads were mapped out and then reduced (also via zmq). This handled around 14 billion rows when I left that job. If you want to talk some more about it, my gmail is the same as my username.
Re:Analyze terabytes of data with just a click of a button by karterk on Tuesday August 21, 2012 @12:25PM (#4413402)
Thanks - I have emailed you.