Re:Neat Parallel Output in Python
by RhysU on Sunday February 25, 2024 @11:54AM
(#39502373)
> you have not explained how to handle progress reporting in gnu parallel.
`parallel --eta` provides progress reporting over all tasks based upon observed completions.
> in tfa the workers are an implementation detail of a wider program.
TFA could write the workers as standalone processes then subprocess out to GNU Parallel.
At that point, one might not even want the Python outer process in this article. I often write little shell wrappers that establish non-trivial GNU Parallel or Make invocations under the hood. Generally, leaning on common workhorses for multiprocessing seems more sensible than rolling my own one-off.
`parallel --eta` provides progress reporting over all tasks based upon observed completions.
> in tfa the workers are an implementation detail of a wider program.
TFA could write the workers as standalone processes then subprocess out to GNU Parallel.
At that point, one might not even want the Python outer process in this article. I often write little shell wrappers that establish non-trivial GNU Parallel or Make invocations under the hood. Generally, leaning on common workhorses for multiprocessing seems more sensible than rolling my own one-off.