Re:GNU Parallel, where have you been all my life?
by cusspvz on Monday August 21, 2023 @08:58AM
(#37208934)
You guys know that in bash you can use `&` to pass a foreground terminal process to the background and then use `wait` to wait for all the session's background process to end, right?
[ Parent
| on HN ]
Re:GNU Parallel, where have you been all my life?
by bduffany on Monday August 21, 2023 @09:27AM
(#37209313)
Yes, and those work well for smaller workloads, but if you just run 1,000,000 commands with `&` in a `for` loop, it will grind your computer to a halt (if the tasks are modestly resource intensive). GNU parallel will let you run those same 1,000,000 tasks but make sure that only (e.g.) 16 of them are running at once. It's not easy to do that in bash.
Re:GNU Parallel, where have you been all my life?
by oniony on Monday August 21, 2023 @09:00AM
(#37208962)
I've been using `&` to run stuff in the background for donkeys, but had no idea about `wait`.
Re:GNU Parallel, where have you been all my life?
by Alifatisk on Monday August 21, 2023 @10:18AM
(#37209874)
Didn’t know about wait
Re:GNU Parallel, where have you been all my life?
by seized on Monday August 21, 2023 @10:25AM
(#37209976)
And that's not really at all comparable to what Parallel can do.... Bash can't do that across thousands of cores on separate machines for example.
Re:GNU Parallel, where have you been all my life?
by remram on Monday August 21, 2023 @12:59PM
(#37212263)
Strong "Dropbox is just rsync it'll never sell" vibes.
Re:GNU Parallel, where have you been all my life?
by da-x on Tuesday August 22, 2023 @01:10AM
(#37218905)
It takes time to notice that if you do _several_ of these background jobs with `&`, you will only get the exit status of the last one when you do `wait`. Errors of the others will be swallowed.
Then you _have_ resort to 'wait <pid>' with the 20 lines of bash coded need to manage all those PIDs. I have a large editor bash snippet just for that.