News for hackers. Stuff that matters.
Re:Command-line Tools can be 235x Faster than a Hadoop Clust... by jake_morrison on Wednesday May 23, 2018 @10:57PM (#17140706)
One fun thing is that xargs supports parallelization.

    -P max-procs
    Run up to max-procs processes at a time; the default is 1.
    If max-procs is 0, xargs will run as many processes as
    possible at a time.  Use the -n option with -P; otherwise
    chances are that only one exec will be done.

    -n max-args
    Use at most max-args arguments per command line.
    Fewer than max-args arguments will be used if the size
    (see the -s option) is exceeded, unless the -x option is
    given, in which case xargs will exit.
So here is a "map/reduce" job which takes the log files in a directory and processes them in parallel on eight CPUs, then combines the results.

    find . -name "*.log" | sort | xargs -n 1 -P 8 agg_stats.py | sort | merge_periods.py
Re:Command-line Tools can be 235x Faster than a Hadoop Clust... by Groxx on Thursday May 24, 2018 @03:29AM (#17141817)
xargs is one of my favorites, it's so easy to split something apart to speed it up by a few multiples.

gotta SSH into 100 machines and ask them all a simple question? xargs will trivially speed that up by at least 10x, if not better, just by parallelizing the SSH handshakes.

Re:Command-line Tools can be 235x Faster than a Hadoop Clust... by adrianN on Thursday May 24, 2018 @12:37AM (#17141135)
And for cases where xargs parallelism isn't flexible enough, there is GNU parallel.
Re:Command-line Tools can be 235x Faster than a Hadoop Clust... by IndustrialJane on Monday May 28, 2018 @11:53PM (#17176355)
But that approach is potentially dangerous. See:

https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_w...