News for hackers. Stuff that matters.
Re:How to speed up massive data analysis by eliminating disk... by barrkel on Saturday January 02, 2010 @07:00AM (#1027122)
It depends. For example, what's the balance between reading and writing, how stream-oriented your processing is, how important is random access between different records, etc.

This approach can be easily augmented too. For example, doing a binary search for a particular line in a text file when you don't have all the lines in RAM is somewhat tedious; it can be made much easier by creating a simple index for the file, consisting of a flat array of the file offset of every line start. That flat array can be stored in a file also; then, both the total number of lines and the contents of a line at any given index are trivial to retrieve.

If you have to handle a small number of updates while still handling lots of reads, then you can use a two-layer approach. Keep a cache of all pending updates in memory in an efficient manner (e.g. hash table), and look up the cache before falling back to the disk; and when writing, both update the cache and write out to an update log, which can be sorted and included in the main store later, when it makes sense.

There are no comments on this comment yet.