The only problem with `tee' is that it's hard to find it unless you know where to look -- took me some 2 years.

<snark warning>

Perhaps the real challenge here is sticking with the team that produces incredible qunatities of code, as they put it, and will rather re-implement tee than find it. I wonder if they implemented own virtual memory with on-demand paging yet ;-)

</snark warning>

Except that if you read the problem statement, you'll see that tee doesn't meet the requirement of 'taking advantage of all available I/O parallelism'. Tee writes to each output file sequentially.

Given that their core business is data-stores that take full advantage of emerging storage technologies (such as flash), this seems like pretty relevant question.

Except that if you do: whatever | tee filea | tee fileb | tee filec | tee filed >/dev/null, you'll get parallelism. This works since tee will write first to stdout, and then to the file you pass it. This may not be "all available" parallelism, but I suspect that it's a decent approximation of "good enough".

I did read the problem statement, but I thought tee was writing to all files in parallel. It's surprising that there seems to be no way to achieve that.

Tee writes in one-thread-per-file level of parallelism; simple experiment:

  $ tee foo.bin bar.bin baz.bin < /dev/zero

writes to all three files `at once' (in small time-slices, actually). Used GNU tee from coreutils-8.10.

Perhaps OP meant several-threads-per-file level of parallelism, where number_threads > number_files?

tee has a simple main loop:

  while not EOF: 
    (synchrounously) read up to BUFSIZE bytes from stdin into buf 
    for each output file: 
      (synchrounously) write buf to file

(source code here: http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tee....)

All of the reads and writes are done through synchronous file apis. I don't see where you are getting one-thread-per-file (or any other kind of parallelism) there.

I am by no means a system programmer (I'd love to learn, no idea where to start), but seeing some sort of either breakdown or a list of implementation challenges to solve this problem would be awesome. I get that the man page describes what it needs to handle, but I don't clearly see every moving piece that this would have to deal with and account for.

I submitted the tee solution as a joke and I got a smiley in response. I was pretty pleased with myself.