Using GNU parallel (YouTube tutorials)

Re:Using GNU parallel (YouTube tutorials) by stephenr on Tuesday May 12, 2020 @06:06AM (#23152332)

We've briefly used parallel for an internal tool when the site went "multisite" (i.e. multiple parallel instances that sometimes need to all do the same thing, such as post-deployment tasks).

We currently have a patch to remove it, and instead use regular task backgrounding and the `wait` shell builtin.

The way output is buffered was not particularly conducive to what we wanted to achieve.

[ Parent | on HN ]

Re:Using GNU parallel (YouTube tutorials) by MachineMartin on Tuesday May 12, 2020 @08:45AM (#23153327)

THE way? It sounds as if GNU Parallel only buffers in one way.

GNU Parallel can handle output in a few ways:

  * --ungroup: With -u output GNU Parallel does not touch the output and output may mix
  * --line-buffer: This buffers a single line from each process.
  * --group: This buffers in files in $TMPDIR. $TMPDIR can point a RAM disk, if you want to buffer in RAM.
  * --sqlworker: This buffers in memory before saving output to an SQL server.

If you are using '&' + 'wait', it pretty much has the same effect as `--ungroup`.

[ Parent | on HN ]