Re:A Million Text Files and a Single Laptop
by coderzach on Tuesday March 08, 2016 @05:02PM
(#11249115)
I don't think that's the case, the default behavior of gnu parallel is to buffer the entire output before outputting. The output will only be interleaved if the `--ungroup` option is used. Parallel also has --line-buffer, which will interleave lines, rather than arbitrary bytes.
Are you suggesting that the command argument to `parallel` is shell that's parsed and reinterpreted by `parallel`?
Buffering the output would make more sense, but the OP's command isn't outputting to stdout, but rather appending directly to a file.
Good point. The command should simply move the >>file outside of the quote, and then it will work.
Re:A Million Text Files and a Single Laptop
by coderzach on Saturday March 12, 2016 @03:05PM
(#11274281)
Oh, yeah, I read it as if the ">>" was outside.
Re:A Million Text Files and a Single Laptop
by yeahrightfam on Tuesday March 08, 2016 @10:00PM
(#11250594)
Yep it's flawed.
That made read the rest of the post with a "meh" attitude instead of my initial "this could be cool" mindset.
If I knew for sure all input files were smaller than PIPE_BUF, I'd do this instead:
That saves you the overhead of firing up a new shell just to reopen a file for append and exec `cat`.