Except that if you read the problem statement, you'll see that tee doesn't meet the requirement of 'taking advantage of all available I/O parallelism'. Tee writes to each output file sequentially.
Given that their core business is data-stores that take full advantage of emerging storage technologies (such as flash), this seems like pretty relevant question.
Re:Did anybody solve this?
by joshcorbin on Friday December 23, 2011 @03:11PM
(#3387061)
Except that if you do: whatever | tee filea | tee fileb | tee filec | tee filed >/dev/null, you'll get parallelism. This works since tee will write first to stdout, and then to the file you pass it. This may not be "all available" parallelism, but I suspect that it's a decent approximation of "good enough".
I did read the problem statement, but I thought tee was writing to all files in parallel. It's surprising that there seems to be no way to achieve that.
Tee writes in one-thread-per-file level of parallelism; simple experiment:
$ tee foo.bin bar.bin baz.bin < /dev/zero
Perhaps OP meant several-threads-per-file level of parallelism, where number_threads > number_files?
tee has a simple main loop:
while not EOF:
(synchrounously) read up to BUFSIZE bytes from stdin into buf
for each output file:
(synchrounously) write buf to file
All of the reads and writes are done through synchronous file apis. I don't see where you are getting one-thread-per-file (or any other kind of parallelism) there.
<snark warning>
Perhaps the real challenge here is sticking with the team that produces incredible qunatities of code, as they put it, and will rather re-implement tee than find it. I wonder if they implemented own virtual memory with on-demand paging yet ;-)
</snark warning>