For heavy lifting check OAR at https://oar.imag.fr/
Never used OAR...
I've used (and installed) PBS, SGE, and SLURM [1]. Most of the clusters I've used recently have all migrated to SLURM. Even though it's pretty feature packed, I've found it "easy enough" to install for a cluster.
What is the sales pitch for OAR? Any particularly compelling features?
Re:Nq – A simple Unix job queue system
by natmaka on Wednesday January 27, 2021 @12:05AM
(#25924522)
(I'm not an expert)
OAR, according to knowledgeable people I've worked with, is robust and extensible.
Re:Nq – A simple Unix job queue system
by cowsandmilk on Wednesday January 27, 2021 @08:18AM
(#25927854)
all of the options can probably claim that...
There are a number of tools in this category, and like others have mentioned, my first try would be Make, if that is an option for you. However, I normally work on HPC clusters, so submitting jobs is incredibly common for me. To keep with that workflow without needing to install SLURM or SGE on my laptop (which I've done before!?!?), my entry into this mix is here: https://github.com/compgen-io/sbs. It is a single-file Python3 script.
My version is setup to only run across one node, but you can have as many worker threads as you need. For what you asked for, you'd run something like this:
This isn't heavily tested code, but works for the use-case I had (having a single-file batch scheduler for when I'm not on an HPC cluster, and testing my pipeline definition language). Normally, it assumes the parameters (CPUs, Memory, etc) are present as comments in a submitted bash script (as is the norm in HPC-land). However, I also added an option to directly submit a command. stdout/stderr are all captured and stored.The job runner also has a daemon mode, so you can keep it running in the background if you'd like to have things running on demand.
Installation is as simple as copying the sbs script someplace in your $PATH (with Python3 installed). You should also set the ENV var $SBSHOME, if you'd like to have a common place for jobs.
The usage is very similar to many HPC schedulers...
etc...