Graphical data exploration and stats with R, python, etc is a beautiful challenge at that scale.
I wound up having a script for users on a jump host that would submit an sbatch job that ran sshd as the user on a random high level port and stored the port in the output. The output was available over NFS so the script parsed the port number and displayed the connection info to the user.
The user could then run a vscode server over ssh within the bounds of CPU/memory/time limits.
I had a co-worker describe it as a giant Linux playground.
Another as ETL nirvana.
* https://developer.lsst.io/usdf/batch.html
* https://epcc.ed.ac.uk/hpc-services/somerville
But they're all over the place, from the James Hutton Institute to London Imperial.
* https://www.cropdiversity.ac.uk
* https://www.imperial.ac.uk/computing/people/csg/guides/hpcom...
And then we aren't even talking about the EPD servers that are amortized in 4 years and can easily become a compute node in the cluster for another 6 (only problem are the bookkeepers who just can't live with post-amortized hardware! What a world!!)
Of course, the organization will pay for some of those eventually, so it's not fully fair to not roll them into the IT costs, but there are also lots of ways that non-profits also don't pay those costs at the same levels that the cloud providers do either due to differences in overall costs, or in providing a lesser level of capability. (As a quick example, cloud providers need extensive physical security for their datacenters. A hospital server needs a locked door, and can leverage the existing hospital security team for free.)
Cloud is great if your need is elastic, or if you have time-sensitive revenue dependent on your calculations. In non-profit research environments, that is often not the case. Users have compute they want done "eventually", but they don't really care if it's done in 1 hour or 4 hours; they have lots of other good work to do while waiting for the compute to run in their background.
[0] https://servicedesk.surf.nl/wiki/spaces/WIKI/pages/30660184/...
Bioinformaticians (among others) in (for example) University Medical Centers won’t get much more bang for the buck than on a well managed Slurm cluster (ie with GPU and Fat nodes etc to distinguish between compute loads). You buy the machines, they are utilized close to 100% over their life time.