Re:Did anybody solve this?
by joshcorbin on Friday December 23, 2011 @03:11PM
(#3387061)
Except that if you do: whatever | tee filea | tee fileb | tee filec | tee filed >/dev/null, you'll get parallelism. This works since tee will write first to stdout, and then to the file you pass it. This may not be "all available" parallelism, but I suspect that it's a decent approximation of "good enough".
I did read the problem statement, but I thought tee was writing to all files in parallel. It's surprising that there seems to be no way to achieve that.
Tee writes in one-thread-per-file level of parallelism; simple experiment:
$ tee foo.bin bar.bin baz.bin < /dev/zero
writes to all three files `at once' (in small time-slices, actually). Used GNU tee from coreutils-8.10.Perhaps OP meant several-threads-per-file level of parallelism, where number_threads > number_files?
tee has a simple main loop:
while not EOF:
(synchrounously) read up to BUFSIZE bytes from stdin into buf
for each output file:
(synchrounously) write buf to file
(source code here: http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tee....)All of the reads and writes are done through synchronous file apis. I don't see where you are getting one-thread-per-file (or any other kind of parallelism) there.
Given that their core business is data-stores that take full advantage of emerging storage technologies (such as flash), this seems like pretty relevant question.