> One thing that I would add is avoid mdadm like the plague. Its tricky, poorly documented and requires a whole bunch of other crap tools to make it work.

Strange, I have exactly the opposite view. I think ZFS is exceptionally well documented and easy to use.

But MDADM is just as simple. What other 'crap tools' are you talking about?

Regarding your RAID60 experience: it could be a bad batch of drives.

Also, I wonder if they had a good solid 24H break-in period with a significant load to weed out bad drives before the array was taken into production.

ZFS's documentation is as good as the docs for expensive storage. (HEllo IBM redbooks, they are _wonderful_)

To use mdadm properly you need lvm as well. Which means dealing with all the rubbish that come with it. (unrecoverable snapshots as a service? that might be fixed now.)

> Regarding your RAID60 experience: it could be a bad batch of drives.

Naaa it was the workload. We had 32 of these server that were largely identical, and bought in two batches. This one (it was actually two) server was running the blackhole fluid sim for interstella. These servers had all been up and running for well over a year.

I should point out that with about 2000 drives for these servers, we would be replacing at least three disks a week. Normally they'd be spread evenly over all servers, it was unusual that this array ate ~7 drives in two weeks, with back to back rebuilds.

Interestingly, the nearline machines which had 4x the drives, was much less likely to loose disks. (however they had a different usage pattern, virtually no random access, and it was one raid controller on the top 60 drive array, then three jbods daisy chained.)

What is the mdadm use-case that requires lvm? I've used mdadm happily for over a decade and never used lvm.

If you want snapshots on XFS, you want lvm, else (AFAICT) you can only make a snapshot while the XFS volume is offline.

Ok clear story, thanks. LVM is a different story. To be frank, my article was focused on the smb / home nas user.

In enterprise environments, it’s a different story.

> One thing that I would add is avoid mdadm like the plague. Its tricky, poorly documented and requires a whole bunch of other crap tools to make it work.

Do you feel that Linux MDRAID is a bad technology, or just a bad UX? I.e., are MDRAID-based appliances (like many NASes)—where all the sharp edges have been smoothed away—still bad in some fundamental way?

Oh no, its a fine technology, its just useless documentation and to make it really work you need LVM, which is just horrific to deal with.

I should say that I've worked proffeshionally with ZFS, mdamd/pv/lvm gluster(ugh) lustre(ok) clustered xfs and GPFS.

Along with a bunch of appliances.

mdadm/pv/lvm is about the bottom three. After gluster and ceph.

I find LVM2 to be easy to work with, but the weird marriage between MD and LVM confuses me, so I just use "hardware" raid beneath LVM.

(For server OS installs that I can recreate trivially, and not for actual data, which lives on a proper SAN because I value my employment)

How come mdadm needs lvm?

Do you mean device mapper?

I don't mind device mapper, for what I used it for, multipath scsi.

mdadm doesn't need lvm. But if you want snapshots, or to have flexible partitions (anything like the flexibility of zfs) you need lvm.

Also for the longest time redhat used to force lvm on you by default. I assume this was to help with "flexibility"

In practice, on virtual machines, or anything back with decent storage, I'd remove the partition table and just run on the block device directly.

mdadm is just fine. YMMV depending on the distro you are using and your knowledge/know-how about storage.

I recommend to try it out with loop devices, play around and build your own opinion about it.