News for hackers. Stuff that matters.
Re:Mapping Out the HPC Dependency Chaos by throw0101c on Friday November 11, 2022 @12:12PM (#33563504)
This is why Spack was created:

> Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments. It was designed for large supercomputing centers, where many users and application teams share common installations of software on clusters with exotic architectures, using libraries that do not have a standard ABI. Spack is non-destructive: installing a new version does not break existing installations, so many configurations can coexist on the same system.

* https://spack.readthedocs.io/en/latest/

> Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. Spack isn’t tied to a particular language; you can build a software stack in Python or R, link to libraries written in C, C++, or Fortran, and easily swap compilers or target specific microarchitectures.

* https://spack.io

Intro presentation from some HPC conferences:

* https://www.youtube.com/watch?v=edpgwyOD79E

* https://www.youtube.com/watch?v=DhUVbroMLJY

Very similar to (home)brew, MacPorts, etc, with more of a focus on HPC so you can have multiple versions of a particular piece of software, made with different compiles and linked against different library version.

Re:Mapping Out the HPC Dependency Chaos by ajdecon on Friday November 11, 2022 @12:18PM (#33563591)
It’s worth noting that one of the paper’s authors is Todd Gamblin, the Spack project lead.
Re:Mapping Out the HPC Dependency Chaos by whinvik on Friday November 11, 2022 @03:42PM (#33566537)
Spack is really awesome. Made my development work during my PhD much easier.
Re:Mapping Out the HPC Dependency Chaos by throw0101c on Friday November 11, 2022 @04:51PM (#33567363)
Todd Gamblin seems to be on HN and is around in this thread:

* https://news.ycombinator.com/item?id=33566807

Re:Mapping Out the HPC Dependency Chaos by 0cf8612b2e1e on Friday November 11, 2022 @12:46PM (#33564022)
I have not personally used Spack, but a similar product in HPC, and I have often wondered why something analogous did not take off for desktop computing. We are finally seeing the rise of Nix, but the experience is currently so bad, I will give it a decade before it starts gaining serious traction. Sooner only if some big platform latches onto it.
Re:Mapping Out the HPC Dependency Chaos by 10000truths on Friday November 11, 2022 @01:29PM (#33564703)
A lot of these selling points sound like features that Nix already has. What does Spack offer that Nix does not? Is it documentation? User experience? Wider support for computing libraries?
Re:Mapping Out the HPC Dependency Chaos by p4ul on Friday November 11, 2022 @01:53PM (#33565030)
I'm not very familiar with Nix, but one rather unique feature of Spack is the concept of combinatorial builds.

Suppose we have a big HPC cluster, and suppose also that this is a fairly heterogeneous cluster in terms of CPU architectures. We might have, for example, 3 different generations of Intel chips. We would also have several different compilers, and several versions of MPI (e.g., OpenMPI, MVAPICH, etc.).

Using Spack, it is trivially simple to build a piece of software with all possible combinations of compiler-architecture-MPI.

Spack also integrates very nicely with LMOD, so managing Spack installs using the LMOD module system is extremely convenient.

I would also argue that the user experience with Spack is phenomenal, but that's just a personal preference.

Re:Mapping Out the HPC Dependency Chaos by tgamblin on Friday November 11, 2022 @04:03PM (#33566807)
Co-author/spack guy/occasional frequenter of Nix fora here.

Aside from what p4ul said above, the main architectural differences between Nix and Spack are: 1. Spack has a dependency solver (the "concretizer"), Nix does not 2. Spack packages are parameterized, and we solve for parameters like build options, compilers, virtual dependencies, etc., so the user can say what tweaks they want on the CLI or in an environment. To do this same with Nix packages you typically need to hack on some nixpkgs. The goal of Spack is to be able to compose and swap libraries, compilers, and other options easily. 3. Spack has more support for external packages (or "impure" packages as the Nix people say). You can point it at an existing installation on your system to use as a dependency.

Hashing installation directories, the store-based installation model described in the paper are things the two have in common.

More on the solver aspect in this other paper that we're also presenting next week: https://arxiv.org/abs/2210.08404

Re:Mapping Out the HPC Dependency Chaos by throw0101c on Friday November 11, 2022 @04:49PM (#33567337)
> What does Spack offer that Nix does not?

Installing specific, older versions of software seem to be a bit of an ordeal with Nix:

* https://lazamar.github.io/download-specific-package-version-...

involving pulling out git and cherry-picking specific commits. Whereas with Spack it is:

* spack install [email protected]

Plus you can do that multiple times times for various values of "x.y.z" and then do a "module load neovim/x.y.z" for whatever version you want.

Multi-versioning is a first-class concept in Spack.

Re:Mapping Out the HPC Dependency Chaos by MontyCarloHall on Friday November 11, 2022 @02:57PM (#33565965)
Why not just containerize your software and run the container on the HPC cluster?
Re:Mapping Out the HPC Dependency Chaos by bayindirh on Friday November 11, 2022 @04:04PM (#33566824)
Disclosure: I’m an HPC admin.

Accessing InfiniBand and GPUs directly become a problem.

Rootless containers are also still at infancy, and creates a lot of problems. You don’t want to give indirect root access via docker group, too.

Re:Mapping Out the HPC Dependency Chaos by maxwell86 on Friday November 11, 2022 @10:44PM (#33569743)
> Accessing InfiniBand and GPUs directly become a problem.

I use nvidia containers on HPC systems every day and accessing NICs, doing RDMA to GPUs, etc. "just works" and performs as well as baremetal. Every time we upgrade our container we verify the new container with a set of benchmarks against both the old one and baremetal.

> You don’t want to give indirect root access via docker group, too.

I don't know of any HPC center using docker though. It does not sound like a good idea because the docker daemon runs as root..

Re:Mapping Out the HPC Dependency Chaos by throw0101c on Friday November 11, 2022 @04:34PM (#33567165)
> Why not just containerize your software and run the container on the HPC cluster?

Docker needs root access which is a big no-no in multi-user environments.

Singularity/Apptainer was developed (with HPC in mind) so that non-admin users could run containerized workloads, and Spack supports creating such workloads:

* https://spack.readthedocs.io/en/latest/containers.html

Re:Mapping Out the HPC Dependency Chaos by bayindirh on Friday November 11, 2022 @04:51PM (#33567357)
Singularity and Apptainer are not silver bullets tho. Both have their limitations and own problems to solve.

They will mature eventually, but bare metal is still the king in HPC realm.

Re:Mapping Out the HPC Dependency Chaos by throw0101c on Friday November 11, 2022 @05:09PM (#33567542)
We have bare metal HPC nodes, but some users want to run containerized apps for their batch jobs: it's not either-or, it's both-and.
Re:Mapping Out the HPC Dependency Chaos by bayindirh on Friday November 11, 2022 @11:05PM (#33569836)
Of course, it’s same for us. I just wanted to point out that these technologies cannot replace bare metal installations for every case as of today.
Re:Mapping Out the HPC Dependency Chaos by bayindirh on Friday November 11, 2022 @04:06PM (#33566849)
Looks interesting. If shows/has advantages over modules, might give it a test drive.
Re:Mapping Out the HPC Dependency Chaos by tgamblin on Friday November 11, 2022 @04:08PM (#33566866)
Spack installs the software and generates the modules (you can customize how). See https://spack-tutorial.readthedocs.io/en/latest/tutorial_mod...