Re:Neat Parallel Output in Python
by globular-toast on Sunday February 25, 2024 @07:32AM
(#39500265)
I prefer using a purpose-built tool like GNU parallel. Parallel's only purpose is to run things in parallel and collect the results together. The advantage is you only have to learn to use it once, rather than learn to do this again and again in all the different languages/tools you might use.
[ Parent
| on HN ]
I can’t help but notice you have not explained how to handle progress reporting in gnu parallel.
Also do you really use parallel from software trying to parallelise its internal workload? Note that in tfa the workers are an implementation detail of a wider program.
> you have not explained how to handle progress reporting in gnu parallel.
`parallel --eta` provides progress reporting over all tasks based upon observed completions.
> in tfa the workers are an implementation detail of a wider program.
TFA could write the workers as standalone processes then subprocess out to GNU Parallel.
At that point, one might not even want the Python outer process in this article. I often write little shell wrappers that establish non-trivial GNU Parallel or Make invocations under the hood. Generally, leaning on common workhorses for multiprocessing seems more sensible than rolling my own one-off.
So you have an example how to spawn gnu parallel from python and displaying progress?