Re:Searching 20 GB/sec: Systems Engineering Before Algorithms
by sitkack on Monday May 12, 2014 @01:48PM
(#7734250)
HDD don't care about opening files. If the FS supported the same operations, I could send grep directly to the block level and have it scan contiguous lists of blocks that cover files I care about. You will get false positives for the overscan (potentially), but you will also achieve the raw rate of the disk (150MB/s for spinning and 500MB/s for solid sate).
So in effect, these big data serialization systems are attempting to lift the file system into userland so they can make just that optimization.
So in effect, these big data serialization systems are attempting to lift the file system into userland so they can make just that optimization.