News for hackers. Stuff that matters.
Re:A Million Text Files and a Single Laptop by sigil on Tuesday March 08, 2016 @04:52PM (#11249031)
Don't do this:

    ls | parallel -m -j $f "cat {} >> ../transactions_cat/transactions.csv"
It only works if all files enumerated by `ls` are smaller than `PIPE_BUF`, which is 4096 bytes on Linux (but was historically only 512 bytes). Writes larger than `PIPE_BUF` are not guaranteed to be atomic. Meaning, if you have input files larger than 512 or 4096 bytes, your output file will have chunks of file A interleaved with chunks of file B.

If I knew for sure all input files were smaller than PIPE_BUF, I'd do this instead:

    find . -type f -print0 | xargs -0 -P8 cat > ../transactions.csv
That saves you the overhead of firing up a new shell just to reopen a file for append and exec `cat`.
Re:A Million Text Files and a Single Laptop by coderzach on Tuesday March 08, 2016 @05:02PM (#11249115)
I don't think that's the case, the default behavior of gnu parallel is to buffer the entire output before outputting. The output will only be interleaved if the `--ungroup` option is used. Parallel also has --line-buffer, which will interleave lines, rather than arbitrary bytes.
Re:A Million Text Files and a Single Laptop by sigil on Tuesday March 08, 2016 @05:18PM (#11249248)
Are you suggesting that the command argument to `parallel` is shell that's parsed and reinterpreted by `parallel`?

Buffering the output would make more sense, but the OP's command isn't outputting to stdout, but rather appending directly to a file.

Re:A Million Text Files and a Single Laptop by gcr on Tuesday March 08, 2016 @09:56PM (#11250576)
Good point. The command should simply move the >>file outside of the quote, and then it will work.
Re:A Million Text Files and a Single Laptop by coderzach on Saturday March 12, 2016 @03:05PM (#11274281)
Oh, yeah, I read it as if the ">>" was outside.
Re:A Million Text Files and a Single Laptop by yeahrightfam on Tuesday March 08, 2016 @10:00PM (#11250594)
Yep it's flawed.

That made read the rest of the post with a "meh" attitude instead of my initial "this could be cool" mindset.