Re:Command-line Tools can be 235x Faster than a Hadoop Clust...
by jake_morrison on Wednesday May 23, 2018 @10:57PM
(#17140706)
One fun thing is that xargs supports parallelization.
So here is a "map/reduce" job which takes the log files in a directory
and processes them in parallel on eight CPUs, then combines the results.
[ Parent
| on HN ]
Re:Command-line Tools can be 235x Faster than a Hadoop Clust...
by Groxx on Thursday May 24, 2018 @03:29AM
(#17141817)
xargs is one of my favorites, it's so easy to split something apart to speed it up by a few multiples.
gotta SSH into 100 machines and ask them all a simple question? xargs will trivially speed that up by at least 10x, if not better, just by parallelizing the SSH handshakes.
Re:Command-line Tools can be 235x Faster than a Hadoop Clust...
by adrianN on Thursday May 24, 2018 @12:37AM
(#17141135)
And for cases where xargs parallelism isn't flexible enough, there is GNU parallel.
Re:Command-line Tools can be 235x Faster than a Hadoop Clust...
by IndustrialJane on Monday May 28, 2018 @11:53PM
(#17176355)
But that approach is potentially dangerous. See:
https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_w...