Re:Appending to a file from multiple processes
by kbenson on Wednesday August 03, 2016 @11:58PM
(#12223295)
Actually, since the data is sequential, I think you can write a very quick and optimized combining process.
E.g.
Open each file and buffer a small amount, such as 4k.
1) Read the first record of each buffer
2) Display the first record of the buffers
3) Read the next record from the buffer you just consumed from, and re-buffer another 4k from it if needed.
4) Go to 2.
Go ahead and change how much you want to buffer from each file depending on what seems to work well and to take advantage of sequential read times if using media that uses that to it's advantage (spinning disks), but I don't see that being very slow. Worst case, you are naively sorting X records each time to find the first, but since you are always consuming them in order, you can easily keep track of the current order and do a dingle comparison per loop after the first.
E.g.
Open each file and buffer a small amount, such as 4k.
1) Read the first record of each buffer
2) Display the first record of the buffers
3) Read the next record from the buffer you just consumed from, and re-buffer another 4k from it if needed.
4) Go to 2.
Go ahead and change how much you want to buffer from each file depending on what seems to work well and to take advantage of sequential read times if using media that uses that to it's advantage (spinning disks), but I don't see that being very slow. Worst case, you are naively sorting X records each time to find the first, but since you are always consuming them in order, you can easily keep track of the current order and do a dingle comparison per loop after the first.