News for hackers. Stuff that matters.
Re:How to speed up massive data analysis by eliminating disk... by tlack on Friday January 01, 2010 @10:20PM (#1026694)
is it really faster to write to a bunch of text files, sort them to a new bigger text file, and then do the insert? seems like a lot of extra steps, all involving a lot of reading and writing..
Re:How to speed up massive data analysis by eliminating disk... by barrkel on Saturday January 02, 2010 @07:00AM (#1027122)
It depends. For example, what's the balance between reading and writing, how stream-oriented your processing is, how important is random access between different records, etc.

This approach can be easily augmented too. For example, doing a binary search for a particular line in a text file when you don't have all the lines in RAM is somewhat tedious; it can be made much easier by creating a simple index for the file, consisting of a flat array of the file offset of every line start. That flat array can be stored in a file also; then, both the total number of lines and the contents of a line at any given index are trivial to retrieve.

If you have to handle a small number of updates while still handling lots of reads, then you can use a two-layer approach. Keep a cache of all pending updates in memory in an efficient manner (e.g. hash table), and look up the cache before falling back to the disk; and when writing, both update the cache and write out to an update log, which can be sorted and included in the main store later, when it makes sense.