I just finished another late night "thanks" to unknown person who decided to use this approach several years ago while setting up 17 servers that I inherited. The /boot partition of these RHEL machines was put on SD cards, which have been gradually dying. So I got to boot the machine (which is 6000 km away) to recovery mode from ISO and re-create the /boot filesystem on HDD (RAID).

Of the 17 servers, only three are now remaining where I haven't had to do this. Not all of these were actual SD card failures, some where done preventively. Still, there have been several SD card failures requiring emergency repair work at inconvenient times. There have been 0 RAID failures requiring similar emergency work on the systems where /boot has been migrated to HDD based RAID.

Of course there is no replacement SD card handy with working /boot filesystem. Actually there is no such thing as "handy" in my case. I can actually reconstruct /boot partition on the HDD faster than anybody could go to the datacenter and replace the SD card. And if there were a replacement SD card, I would need to keep it up to date manually every time the kernel or initrd are updated.

I never want to see this kind of setup again, and frankly it felt insane the first time I saw it.

Of course I should have migrated all of the machines off the SD cards by now, but my excuse is that maintaining Linux on these machines is not really my responsibility (it is nobody's responsibility apparently, although many people are interested in keeping these systems online).

When I die I want to have a SD-card shaped tombstone.

This was a trend for a minute back in the late 00's. Any system that was

  1.HA
  2.Had an OS that would get fully loaded into memory (think ESX).

The idea was that SD-cards take less power and where cheap enough you could have a whole gang of spares with images ready to just drop in place. I'm not saying it was a good idea, just that I've encountered this more than once.

So you discovered the well known problem in "IoT" that SD cards should never be used for filesystems that are online 100% of the time?

I think the parent views on storage is still valid. Do not boot from the RAID volume where you store many terabytes.

I usually have a small mirror SSD for the OS (like 128G or max 256G). Those drives are super cheap, easy to replace.

For the /data I usually use ZFS or mdadm + xfs. Depending on the use case (for example do you want to add more capacity later? ) you can decide.

Using SDcards for server filesystem is negligent at the very least.

flash dying after a few years is not exactly weird or unexpected, regardless of the form factor

I don't get why you need to do so. RAID is not a substitute of backups, and shouldn't used that way. Meaning that of course you have a backup of the data that you can recover from.

RAID is used for two things:

1. improving the performance of slow disks (at least the read performance) 2. having 1/2 disk fail with your system remaining completely usable, till you replace the faulty disk as soon as possible.

The second point is fundamental to me: there shouldn't be any disruption of service whatsoever, meaning that only the sysadmin should notice the fault (beside maybe a reduced performance of the system since you have 1 less drive). Database transaction that were in progress when the disk did break shouldn't fail, writes/reads on the FS shouldn't fail, the only thing that should happen is an alarm triggered in the monitoring system to inform that a disk needs to be changed as soon as possible.

Having a RAID with manual recovery... it means that you could end up a Saturday evening in front of a computer to bring a system back online, and still some data corruption may have happened.

Regardless of what you think, many people keep a separate boot and data disk and don't use RAID on the boot disk. I never said anything about it being used for backups nor did I said anything about service outages. I think my point- about what other people do since I don't use RAID- is more about the choices different people make in terms of trusting their boot volume's recoverability in times of crisis.

3. Prevent bit-rot w/ software RAID. Running ZFS, btrfs et al. on a single disk can detect bit-rot but not repair it.

By default the metadata profile is 'DUP' (i.e. 2 copies on one device), the same can be done for data but this reduces the usable capacity. On normal HDD or SSD this should not be needed but having both data and metadata DUP/DUP has been useful on raspberry pi with the micro SD card storage. It's not perfect but increases the chances to get the data back if the card is partially damaged due to power spikes.

I have wondered about this myself - does keeping 2 copies of the same data on a single flash-based drive actually increase reliability? Or is the flash controller going to end up combining the two writes into the same block?

That seems like a very hacky way to accomplish something that should just work. I mean, it's better than nothing, but it seems far from ideal.

If it's your own personal box, and you're okay with that fiddling, fine. But if people are supposed to get work done while the admin gets a new disk, it's not great.

There's pretty solid reasons why you might choose to have a very simple RAID1 set for a boot volume-- because you want to be sure to boot even if the huge array is degraded, and because you want faster I/O for the OS install, etc...

The way I used to do it (back then getting grub/initrd raid aware wasn't really doable - not sure about now) with mdadm was to use RAID1 on the boot volume such that a disk from a broken mirror was still usable by itself. Other volumes were RAID 5 or 10 etc.

How would a mirrored boot work in practice? Normally the first bootable partition gets loaded if it’s corrupt, it would keep failing at the same bad sectors.

Would it randomly pick one of the bootable disks? That way you have an n-1/n chance of avoiding the bad disk?

Back then, grub and initrd didn't understand mdadm so would be set to boot off one of the boot vols as a raw disk. Once initrd handed over to the root vol, it would understand mdadm properly.

If a disk with a boot vol failed, it would have a 50/50 chance of still booting depending on which one failed. You'd create a backup grub entry to boot off the other one manually.

Not as transparent as hardware RAID, and maybe now grub is aware enough to auto handle mdadm? I don't know - I haven't done bare metal mdadm for a decade or so.

"How would a mirrored boot work in practice?"

The same way it has worked for the past 30+ years: the firmware boots from first designated boot device; if that device isn't bootable, it moves on to the next one. The next one boots since it's part of a mirror and has all the necessary data to do so, and in the rare case where the device is "half bootable" one would simply intervene and select the next good bootable device manually.

That's fair enough for a system that isn't critical, but you can almost guarantee that the disk failure will happen at the most inconvenient time. I'd rather have a couple of cheap disks and MD software RAID so that the system boots as long as one of the drives hasn't failed. Then you can replace/fix the failing disk at a convenient time.

Back when solaris wasn't using zfs yet, we were using the live upgrade utility to duplicate the main boot environment on a local disk every night.

Machines booting from SAN had only one local disk, others would have 3 local disks (small and cheap ones because application data was on SAN anyway).

Main raid would do the job for availability in term of disk failure, 3rd bootdisk did the job in case of user erroror data corruption on the main boot env. It saved our asses a few times, bringing apps back online quickly and saving us from reinstalling or fixing stuff from a tenporary live environment.

Why ? mdadm RAID1 is quick and easy solution for that...

why_not_both.jpg

I have 2 NixOS-based NASes that run ZFS. each one has 3 equal-sized 256gb SSDs in addition to the pile o' spinning rust.

each SSD has a small UEFI boot partition, then the rest of the space is cut in half.

the root filesystem is a 3-way ZFS mirror of the first half of the SSDs. the second half of each SSD is another 3-way mirror, this time as the "special" / metadata device for the main hard-drive-backed zpool.

any of the 3 SSDs can fail, and it will boot up and mount the storage perfectly fine. I could also easily upgrade to larger SSDs, in-place and with minimal downtime (zero downtime, if my case had hot-swappable SSD bays)

it would work just as well with only 2 SSDs, but the incremental cost of a 3rd is small enough relative to the whole that I went for it.

I haven't used raid for an OS or boot partition in years. We focus on provisioning and HA. Data goes on a raid an that's it.

Servers go boom, refuse to boot, or even get trigger a few alarms. Disk is replaced, server is booted, and provisioning paves the entire thing faster than you'd be able to debug.

The way I like to do it is with mirroring of the nvme storage and then lvm snapshots on top of it. Done correctly you get boot off either of them but they're kept in sync automagically by the OS. then you snapshot the boot volume regularly and copy those to some kind of archival storage for backups.

That lets you then even hotswap them if you need to while keeping the root filesystem workable.

Plain old USB thumb drives are pretty common for this

I can't knock it, but I imagine I'd be yelling for at least RAID1 if I had to wait for a replacement.

There are even some servers with hardware RAID1 on 2 microsd cards