News for hackers. Stuff that matters.
Re:Parallel processing with Unix tools by philsnow on Saturday September 23, 2017 @12:23PM (#15320389)
I do this pattern a lot

    for x in $(seq 1 10); do long_process file${x} & done; wait
It doesn't take care of not saturating my cpu or anything; when I need to care about that then I try to remember how to use parallel.
Re:Parallel processing with Unix tools by enriquto on Saturday September 23, 2017 @12:29PM (#15320421)
it's actually easier using parallel :

        for x in ...; do echo long_process file$x ; done | parallel -j 8

EDIT: and in your case, it is even easier with xargs, e.g.:

        ls files* | xargs -n 1 -P 8 long_process
Re:Parallel processing with Unix tools by sigjuice on Saturday September 23, 2017 @04:12PM (#15321474)
Wouldn't this fail if your file names have white space?
Re:Parallel processing with Unix tools by Anthony-G on Saturday September 23, 2017 @07:24PM (#15322332)
Parsing `ls` is never a good idea[1]. A more robust way to use globbing in a POSIX shell would be something like this:

    printf "%s\0" files* | xargs -0 -n 1 -P 8 long_process
1. http://mywiki.wooledge.org/ParsingLs
Re:Parallel processing with Unix tools by enriquto on Sunday September 24, 2017 @01:40AM (#15323497)
I disagree with the article that you linked. Filenames are variable names that you get to choose. Half of the game is won by chosing them wisely, so that they are convenient to use.
Re:Parallel processing with Unix tools by flatfilefan on Sunday September 24, 2017 @10:45AM (#15324894)
I use this template to deal with reasonable filenames:

    find ... -print0|parallel --gnu -X -0 -n 1 your_command "{}" \;
And sometimes the names are so screwed that they have to be renamed first, then this can give you a reasonable new name

    $(echo $(basename "{}")|tr '[[:blank:]]' '_'| tr -cd '\/.[[:alnum:]]_-' )
Re:Parallel processing with Unix tools by enriquto on Sunday September 24, 2017 @01:37AM (#15323487)
yes, and rightfully so. If you don't use silly filenames you can write simpler scripts.
Re:Parallel processing with Unix tools by kwhitefoot on Sunday September 24, 2017 @03:31AM (#15323734)
You don't always have the luxury of choosing the names that the files have.
Re:Parallel processing with Unix tools by enriquto on Sunday September 24, 2017 @04:11AM (#15323839)
I actually do.

In the rare cases where I have to work with odd filenames, it is easier to rename those files than to change my clean scripts.

Re:Parallel processing with Unix tools by SamJson on Sunday September 24, 2017 @06:56AM (#15324251)
Ahh, I see a sysadmin with users! :)
Re:Parallel processing with Unix tools by kwhitefoot on Monday September 25, 2017 @06:57AM (#15329400)
Well, ex-sysadmin for many years now. But yes, real world and all that.
Re:Parallel processing with Unix tools by SamJson on Sunday September 24, 2017 @03:45AM (#15323777)
I would do:

    seq 10 | parallel long_process file{}
Or simply:

    parallel long_process ::: file*