Yes, nodes have NVMe SSD local scratch space for fast compute needs. Large datasets reside in central storage. Smaller chunks are usually brought to local scratch to let compute jobs do their thing. Results go back to central storage again.
I built it for our needs based on freely available docs and community support. The server runs FreeBSD + ZFS + NFS and nothing else. Snapshots from this primary storage gets shipped over to a secondary location that has a slow mirror(similar setup but with cheap SATA disks). Our need is only minor geographical data durability and not lower downtime DR.
Experience: beats most commercial stuff we have run so far. Minimal setup of FreeBSD, ZFS, and NFS runs buttery smooth with almost no operational needs. I run upgrades on OS after it has been in community for a while, and it has only been smooth. We didn’t find a need to buy specialist support as our setup is minimal, and workload not any crazy, and availability targets not so strict.
Performance wise this setup far exceeded our expectations in all our usecases. We do have 768 GiB of memory for this that ZFS happily makes use of. ARC is about 60-70% of mem most of the time.
I will happily build the same again should there be a need for storage.
So, we make do with what we have. We have three similar storage configs(and three mirrors as slow backups). We carry spare memory sticks, Optane drives, disks, and power supply for minimising down time due to hardware failures. If something were to go wrong for which we don't have spares, yes, the storage will be offline until we can fix it.
Planned updates? - we have two scheduled downtimes in a year. All patches to these machines happen at that time. These machines are strictly internal and don't have internet access except for those planned maintenance days.
e: Not doing market research here, I'm just seeing (sometimes, feeling) all the pain involved with cluster filesystems and wondering how your solution stacks up.