ole_tange

Examples simulated with GNU Parallel:

    simple_gpu_scheduler --gpus 0 1 2 < gpu_commands.txt
    parallel -j3 --shuf CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' < gpu_commands.txt

    simple_hypersearch "python3 train_dnn.py --lr {lr} --batch_size {bs}" -p lr 0.001 0.0005 0.0001 -p bs 32 64 128 | simple_gpu_scheduler --gpus 0,1,2
    parallel --header : --shuf -j3 -v CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =}' python3 train_dnn.py --lr {lr} --batch_size {bs} ::: lr 0.001 0.0005 0.0001 ::: bs 32 64 128

    simple_hypersearch "python3 train_dnn.py --lr {lr} --batch_size {bs}" --n-samples 5 -p lr 0.001 0.0005 0.0001 -p bs 32 64 128 | simple_gpu_scheduler --gpus 0,1,2
    parallel --header : --shuf CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1; seq() > 5 and skip() =}' python3 train_dnn.py --lr {lr} --batch_size {bs} ::: lr 0.001 0.0005 0.0001 ::: bs 32 64 128

    touch gpu.queue
    tail -f -n 0 gpu.queue | simple_gpu_scheduler --gpus 0,1,2 &
    echo "my_command_with | and stuff > logfile" >> gpu.queue

    touch gpu.queue
    tail -f -n 0 gpu.queue | parallel -j3 CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' &
    # Needed to fill job slots once
    seq 3 | parallel echo true >> gpu.queue
    # Add jobs
    echo "my_command_with | and stuff > logfile" >> gpu.queue
    # Needed to flush output from completed jobs 
    seq 3 | parallel echo true >> gpu.queue

Fun fact: GNU Parallel used to be a wrapper for making Makefiles: https://www.gnu.org/software/parallel/history.html

Also see this if you have a system without GNU Parallel: http://oletange.blogspot.dk/2013/04/why-not-install-gnu-para...

--line-buffer may or may not give additional speed up.

--pipe is well know for being slow.

Try --pipe-part instead:

    parallel -a ngrams.tsv --pipe-part --block -1 awk -f map.awk |
      awk -f reduce.awk

> Granted, it's always possible that the parallel on $PATH might not be the one you're thinking of, but... Whatevs.

For exactly this reason, GNU Parallel has the option --minversion.

So in your script you put:

parallel --minversion 20140722 || exit

if the rest of your script depends on functionality only present from version 20140722.

If the parallel in the $PATH is not GNU Parallel it will fail (and thus exit). If it _is_ GNU Parallel it will fail if the version is < 20140722 and succeed otherwise.

> Same syntax applies.

Except that is not entirely true: For instance according to the author the '{= perl expression =}' will probably never be supported.

(Full disclosure: I am the author of GNU Parallel. I fully support building other parallelizing tools, but to avoid user confusion, I would recommend calling them something other than 'parallel' if they are not actually compatible with GNU Parallel).

Rust-parallel _is_ fast, and there is clearly a niche here, that GNU Parallel is unlikely to fill: By design GNU Parallel will never require a compiler; this is so you can use GNU Parallel on old systems with no compilers (Think an old, dusty AIX-box that people have forgotten the root password to). This design decision limits how fast GNU Parallel can be compared to compiled alternatives.

But the main problem with rust-parallel is that it is not compatible with GNU Parallel (and according to the author, it probably never will be 100% compatible). If you use rust-parallel to walk through GNU Parallel's tutorial (man parallel_tutorial) you will see it fails very quickly.

(Full disclosure: I am the author of GNU Parallel. I fully support building other parallelizing tools, but to avoid user confusion, I would recommend calling them something other than 'parallel' if they are not actually compatible with GNU Parallel. History has shown that using the same name will lead to a lot of unnecessary grief: e.g. GNU Parallel vs. Parallel from moreutils).

The page with differences to other tools will be moved to 'man parallel_aternatives' in next version. It made sense when the section was short, but it has grown so big that it can make it harder to navigate.

Thanks for input.

(The trick to find the examples: LESS=+/EXAMPLE: man parallel )

Except for GNU Parallel --pipe. When you use '<' what you typically mean is '--pipepart -a file'.

Sure do. Makes you feel that the future is now!

Did you start by reading the "Reader's guide" which is presented before the first option is introduced in the man page?

Do you have an example of "piping next to process invocation"? It sounds interesting.

Just be aware that Stderr will contain stuff you did not ask for: https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...

The syntax difference is small for simple tasks.

A good start is looking at the section 'Remote execution' in 'man parallel_tutorial'.

Spoiler: You can have GNU Parallel kick the machines if the fail, and rerun the job on another machine.

The options section is no doubt for people, who know the tool. But the huge section with examples is not.

Did you read the "Reader's guide" which is presented before the first option in the man page?

A lot of effort has gone into the documentation of GNU Parallel: Intro videos, a one hour tutorial, tons of examples, a man page on the design decisions behind the code, and even a "Reader's guide" in the man-page.

Can you be a bit more specific why you believe it is a "brick-wall-in-your-face"? Did you follow the "Reader's guide", which is before even the first option is introduced?

Can I ask to to run these and then comment?

LESS=+/EXAMPLE: man parallel

LESS=+/Reader.s man parallel

man parallel_tutorial

Fun fact: GNU Parallel was originally a wrapper script that used `make` for the parallelization: https://www.gnu.org/software/parallel/history.html

"inside" = temporary files - not RAM. See comparison on https://www.gnu.org/software/parallel/parallel_design.html#B...

I hope you wont take this the wrong way, but can you elaborate on why you feel Tollef's parallel (from moreutils) is better than GNU Parallel?

Using Makefiles, however, depends on the results being files. If that is not the case, make cannot tell how far you got.

GNU Parallel has --joblog and can continue from where it left off or retry all failed jobs again.

Bigger list: https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...

From: https://www.gnu.org/software/parallel/history.html

> parallel dates back to around the same time. It was originally a wrapper that generated a makefile and used make -j to do the parallelization.

Hitachi already made your theme song: https://www.youtube.com/watch?v=xb_PyKuI7II :)

> a version in C that I think was first released before GNU parallel

While this is technically correct, it is misleading: GNU Parallel existed before it became GNU. See details on: https://www.gnu.org/software/parallel/history.html

And it is exactly the hacking part that GNU Parallel tries to help with: A lot of the helper functions in GNU Parallel could be done by expert users (--nice, --tmux, --pipepart, env_parallel, --compress, --fifo, --cat, --transfer, --return, --cleanup, --(n)onall).

But non-expert users will invariably make mistakes (e.g. get quoting wrong, not getting remote jobs to die if the controlling process is killed, or re-scheduling jobs that were killed by --timeout), and why not just have small wrapper scripts built into GNU Parallel that are well-tested, so the non-expert users can enjoy the same stability as the expert users?

One of the reasons for the high line number count is the decision not to depend on modules that are not part of the core of Perl 5.8.

Quite a few of the lines are to deal with different flavours of operating systems.

The benefit of this is that by copying the same single file you can have GNU Parallel running on FreeBSD 8, Centos 3.9, and Cygwin.

It does: Rsync.

GNU Parallel is extremely sluggish because it does all sort of different things behind your back: It buffers output on disk (so the from different jobs are not mixed and you are not limited by the amount of RAM - it will even compress the buffering if you are short on disk space), it checks if the disk is full for every job (so you do not end up with missing output), it gives every process its own process group (so the process with children can be killed reliably with --timeout and --memfree), and a lot of other stuff.

It lets you code your own replacement string (using --rpl), and lets you make composed commands with shell syntax:

    myfunc() { echo joe $*; }
    export -f myfunc
    parallel 'if [ "{}" == "a" ] ; then myfunc {} > {}; fi' ::: a b c

It does not need a special compiler, but runs on most platforms that have Perl >=5.8. Input can be larger than memory, so this:

    yes `seq 10000` | parallel true

will not cause your memory to run full.

You can read a lot more about the design in `man parallel_design` and see the evolution of overhead time per job compared to each release on: https://www.gnu.org/software/parallel/process-time-j2-1700MH...

In other words: Treat GNU Parallel as the reliable Volvo that has a lot of flexibility and will get the job done with no nasty corner case surprises.

It is no doubt possible to make a better specialized tool for situations where the overhead of a few ms per job is an issue and where you neither need brakes, seatbelts nor airbags. xargs is an example of such a tool, and you can have both GNU Parallel and xargs installed side by side.