Re:Ruby concurrency explainined simply
by zepolen on Wednesday February 23, 2011 @07:47AM
(#2253883)
What you described is still not going to be faster than just using the file system. It's just an unnecessary layer on top of it.
You should probably test that assumption.
I want to, but it's hard to make an artificial test for this. You need to have a dataset that's larger than available ram and a read queue that follows real life access patterns but also uses different enough data to force disk reads to happen.
If the theory checked out I would go ahead and implement it - but the theory says it would be slower.
The basic design of the system is that the readers register an event handler with the writer and when the writer finishes writing a file it updates the readers with the location of the file in the ZIP file. Once you have that information it's a simple read call to the ZIP file.