https://aravindh.net/post/sysadmin/
> What form do these jobs have?
Mostly batch jobs written as bash scripts. Occasionally, some users run singularity containers. But, all through SLURM.
> How do you manage workloads?
SLURM
> How do you manage resources?
As a sysadmin, my inventory is via Ansible. All activity on servers happen via Ansible only.
> Do users have quota for compute and storage?
Yes, Users typically can run 72 cores, 512 GiB of mem at a time. Rest is queued until resources are released. Disk quota is only for home directories - 400GiB(enforced by ZFS refquota).
> Do you use GPUs? If so, how do you deal with malfunction?
No, weirdly our workloads(genetics and genomics) don’t fit the GPUs very well, as they are sparse matrix walks with wide precision floats. But, we plan to try for some other stuff soon.
> Is this a distributed processing?
No. Jobs run one node at a time.
> Are the machines heterogeneous?
Yes, inteL based servers all the way from Haswell to Cascade lake.
> What do you use for that cluster?
SLURM
> What if a job requires dependencies?
Taken care of by SLURM.
> Do you create a compute environment on the fly or do all jobs have the same dependencies and these don't change much?
I guess you mean the software libraries and tools that jobs use? If so, our central software repo is NFS mounted on all compute nodes. Users can install things they need if admin priv is not needed. If not either Singularity or email to me.
> How do you do data governance?
This is the painful and human oriented task. We lock down data transport to outside world, educate the users about data policies and then spend a lot of time looking at data flows with hope.
> Is the data read only? Do you have an API to fetch the data from the job code?
Sorry, I could not understand this question. Do you mean metadata about A job?
To manage runtime dependencies we use cvmfs.
A read only distributed filesystems, kinda like a very specific NFS for software distribution.
If that is never been a problem, cvmfs it is quite stable and very used in our field.
Now, we thought of several ways of solving this. We have a branch that uses Kubernetes but this is not the one that is deployed.
We know about SLURM and this is something we will support for a coarse granularity. i.e: notebook level jobs. This is a common scenario and we'll need it for our AppBook (we allow users to turn a notebook into an application with one click and we automatically generate the form fields for the features, and the API endpoint for the model).
However, I'm interested in finer granularity computation; as in: I want a job for the cell. This involves looking into Jupyter kernels to circumvent the front-end<-->kernel disconnection (we'll have to do that for another use case in low internet bandwidth scenario).
>Sorry, I could not understand this question. Do you mean metadata about A job?
How do your users upload, and manage data access and is the data mutable and versioned. What version of the code ran on which version of the data, and who changed that data, etc.
How do users access the data from a Jupyter notebook? Is the data in different object storages? Do you proxy the requests and handle access, things like that.
With respect to the original questions -- I've investigated running a full SLURM cluster on the cloud and it just never seems worth the effort. For larger clusters than mine, maybe, but then when you start to hit the levels you're talking about, I just don't see the point in moving to the cloud. It would be hard to hit that sweet spot where the costs would make sense. Amazon even has a series on scaling a SLURM cluster with AWS EC2 provisioning, but it just seemed like more work than was justified for us.
There's a pretty interesting NSF-wide project for managing clusters in a more commoditized way as part of https://www.xsede.org. You might describe it as a "private heterogeneous almost-cloud" in its goals? That might be saying a bit much.
Density, latency, storage throughput, etc. were in favor of DIY (plus the pricey professionals to run it) rather than cloud offerings. When I was there (2016) I did some basic math for being able to use a cloud provider for some of our lighter workloads when the local cluster was loaded down. Astronomical without a contract, which is quite the thing to set up, etc.
Worth noting that while they do alright, NAU (first link) is hardly a top-tier university with bleeding edge technical requirements.
What form do these jobs have?
How do you manage workloads?
How do you manage resources? Do users have quota for compute and storage?
Do you use GPUs? If so, how do you deal with malfunction?
Is this a distributed processing? Are the machines heterogeneous? What do you use for that cluster?
What if a job requires dependencies? Do you create a compute environment on the fly or do all jobs have the same dependencies and these don't change much?
How do you do data governance? Is the data read only? Do you have an API to fetch the data from the job code?