News for hackers. Stuff that matters.
Re:Parallel processing with Unix tools by enriquto on Saturday September 23, 2017 @06:57AM (#15319186)
do not forget about "make", that will traverse a user-supplied tree of filenames in parallel
Re:Parallel processing with Unix tools by dahart on Saturday September 23, 2017 @09:35AM (#15319653)
And make -j will execute the rules & build targets in parallel. Once I built a general batch parallel system using make that could pause & resume parallel jobs by using each job's log file as the make target. Then I discovered gnu parallel and scrapped my project.
Re:Parallel processing with Unix tools by SamJson on Sunday September 24, 2017 @06:55AM (#15324249)
Fun fact from https://www.gnu.org/software/parallel/history.html

> [GNU Parallel] was originally a wrapper that generated a makefile and used make -j to do the parallelization.

Re:Parallel processing with Unix tools by dahart on Sunday September 24, 2017 @01:26PM (#15325600)
Ha! Crazy, I didn't know that, thanks for the pointer. I recall having some issues with scaling to very large jobs, when I got lists of targets too long for make to deal with. (I think... I could be mis-remembering the details, but something in my pipeline would fail with really big batches.) Maybe parallel split away from make when it ran into similar issues, I wonder.
Re:Parallel processing with Unix tools by gnufx on Saturday September 23, 2017 @02:20PM (#15320959)
Dependencies in batch resource managers for distributed parallel jobs (or simpler ones) are standard practice. https://arc.liv.ac.uk/SGE/htmlman/htmlman1/qmake.html is an old adaptation of GNU Make which works within a job.
Re:Parallel processing with Unix tools by anitil on Sunday September 24, 2017 @07:29PM (#15327068)
Gah story of my life - hours/days of grueling work trying to solve a problem, then someone say 'why don't you just use $tool_that_does_exactly_what_you_want_but_better?'. Code goes in the bin.
Re:Parallel processing with Unix tools by pixelbeat on Sunday September 24, 2017 @06:04PM (#15326732)
Very good point. I added a whole section to the article, implementing the counting lines example with `make -j`, which performs just as well as `xargs -P`
Re:Parallel processing with Unix tools by pletnes on Saturday September 23, 2017 @07:28AM (#15319273)
Or any job in general. Also, make 4.x comes with the handy -O flag which makes the output seem "serial" instead of intermixed between processes.