Of the 17 servers, only three are now remaining where I haven't had to do this. Not all of these were actual SD card failures, some where done preventively. Still, there have been several SD card failures requiring emergency repair work at inconvenient times. There have been 0 RAID failures requiring similar emergency work on the systems where /boot has been migrated to HDD based RAID.
Of course there is no replacement SD card handy with working /boot filesystem. Actually there is no such thing as "handy" in my case. I can actually reconstruct /boot partition on the HDD faster than anybody could go to the datacenter and replace the SD card. And if there were a replacement SD card, I would need to keep it up to date manually every time the kernel or initrd are updated.
I never want to see this kind of setup again, and frankly it felt insane the first time I saw it.
Of course I should have migrated all of the machines off the SD cards by now, but my excuse is that maintaining Linux on these machines is not really my responsibility (it is nobody's responsibility apparently, although many people are interested in keeping these systems online).
When I die I want to have a SD-card shaped tombstone.
1.HA
2.Had an OS that would get fully loaded into memory (think ESX).
I think the parent views on storage is still valid. Do not boot from the RAID volume where you store many terabytes.
I usually have a small mirror SSD for the OS (like 128G or max 256G). Those drives are super cheap, easy to replace.
For the /data I usually use ZFS or mdadm + xfs. Depending on the use case (for example do you want to add more capacity later? ) you can decide.
Using SDcards for server filesystem is negligent at the very least.
RAID is used for two things:
1. improving the performance of slow disks (at least the read performance) 2. having 1/2 disk fail with your system remaining completely usable, till you replace the faulty disk as soon as possible.
The second point is fundamental to me: there shouldn't be any disruption of service whatsoever, meaning that only the sysadmin should notice the fault (beside maybe a reduced performance of the system since you have 1 less drive). Database transaction that were in progress when the disk did break shouldn't fail, writes/reads on the FS shouldn't fail, the only thing that should happen is an alarm triggered in the monitoring system to inform that a disk needs to be changed as soon as possible.
Having a RAID with manual recovery... it means that you could end up a Saturday evening in front of a computer to bring a system back online, and still some data corruption may have happened.
If it's your own personal box, and you're okay with that fiddling, fine. But if people are supposed to get work done while the admin gets a new disk, it's not great.
Would it randomly pick one of the bootable disks? That way you have an n-1/n chance of avoiding the bad disk?
If a disk with a boot vol failed, it would have a 50/50 chance of still booting depending on which one failed. You'd create a backup grub entry to boot off the other one manually.
Not as transparent as hardware RAID, and maybe now grub is aware enough to auto handle mdadm? I don't know - I haven't done bare metal mdadm for a decade or so.
The same way it has worked for the past 30+ years: the firmware boots from first designated boot device; if that device isn't bootable, it moves on to the next one. The next one boots since it's part of a mirror and has all the necessary data to do so, and in the rare case where the device is "half bootable" one would simply intervene and select the next good bootable device manually.
Machines booting from SAN had only one local disk, others would have 3 local disks (small and cheap ones because application data was on SAN anyway).
Main raid would do the job for availability in term of disk failure, 3rd bootdisk did the job in case of user erroror data corruption on the main boot env. It saved our asses a few times, bringing apps back online quickly and saving us from reinstalling or fixing stuff from a tenporary live environment.
I have 2 NixOS-based NASes that run ZFS. each one has 3 equal-sized 256gb SSDs in addition to the pile o' spinning rust.
each SSD has a small UEFI boot partition, then the rest of the space is cut in half.
the root filesystem is a 3-way ZFS mirror of the first half of the SSDs. the second half of each SSD is another 3-way mirror, this time as the "special" / metadata device for the main hard-drive-backed zpool.
any of the 3 SSDs can fail, and it will boot up and mount the storage perfectly fine. I could also easily upgrade to larger SSDs, in-place and with minimal downtime (zero downtime, if my case had hot-swappable SSD bays)
it would work just as well with only 2 SSDs, but the incremental cost of a 3rd is small enough relative to the whole that I went for it.
Servers go boom, refuse to boot, or even get trigger a few alarms. Disk is replaced, server is booted, and provisioning paves the entire thing faster than you'd be able to debug.
That lets you then even hotswap them if you need to while keeping the root filesystem workable.
I can't knock it, but I imagine I'd be yelling for at least RAID1 if I had to wait for a replacement.
Every discussion about btrfs immediately escalates to how the btrfs RAID support develops. But btrfs is useful on its own without that mode.
Should you wish to put a btrfs file system on software RAID (DM), you can still do that. The btrfs RAID modes are something else, where you cut the DM driver out of the picture. That's not going to be as well tested as plain old DM, even if it works, and it doesn't offer any functionality beyond plain DM RAID.
If the use case is storing real data, you want the data storage layer to be as well tested as possible. There's also more to software RAID than the block device. You want a mature resilvering daemon on a well tested schedule, and you need working monitoring to alert you of issues. RAID without monitoring will only delay the inevitable.
wikipedia: The device mapper is a framework provided by the Linux kernel for mapping physical block devices onto higher-level virtual block devices. It forms the foundation of the logical volume manager (LVM), software RAIDs and dm-crypt disk encryption, and offers additional features such as file system snapshots.[1]
Device mapper works by passing data from a virtual block device, which is provided by the device mapper itself, to another block device. Data can be also modified in transition, which is performed, for example, in the case of device mapper providing disk encryption or simulation of unreliable hardware behavior.
To expand a typical RAID setup, you have to essentially fail every single drive in the array and replace them one by one with new identical drives. Expanding an array of N drives requires N resilvers which not only takes a long time but increases the chance of actually failing those drives due to massive I/O load. It's easier to build a second array and copy the data over. This combined with the fact you have to buy all the hardware up front makes it cost prohibitive for home users. You can't buy drives one at a time and expand capacity as needed.
I have used the FUSE port of ZFS to write only one member of a mirroset, then upon mounting both members elsewhere, the stale mirror was very quickly resilvered, so ZFS was able to determine only the blocks needing to be refreshed.
In btrfs, I understand that this requires a rebalance, which will read/write every used portion of the filesystem.
ZFS is far friendlier in a crisis.
"We'll even manually trigger a scrub—a procedure that storage admins generally understand to look for and automatically repair any data issues... even though we manually initiated a scrub and let it finish, our array is still inconsistent and even outright non-mountable, because it ran for a little while without a disk and then that disk was re-added. The command that we were supposed to run was btrfs balance—with both drives connected and a btrfs balance run, it does correct the missing blocks, and we can now mount degraded on only the other disk, /dev/vdc. But this was a very tortuous path, with a lot of potential for missteps and zero discoverability."
https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...
There is a potential gotcha with parity being wrong following a crash or powerfail, the so-called write hole. Scrub doesn't check parity, and parity isn't checksummed. The write hole though is arguably two parts: wrong parity and propagated wrong reconstruction.
In the usual write-hole case, both happen silently. Wrong parity could happen after a power fail, crash, misdirected or torn write - whether the array is functioning normally or degraded. However, wrong reconstruction only happens if there's a failure to read a data strip (bad sector or full device failure).
On btrfs, only the wrong parity being written is possible. Upon reconstruction from bad parity, the resulting data is compared to csum and will fail, thus propagation doesn't happen.
To cause parity to be recomputed and rewritten, yes you need to do a full balance. But among all the block group profiles, that's only raid5 and raid6. Single, DUP, raid1, raid1c3, raid1c4, raid10 aren't affected and a scrub does reallocate missing blocks on mirrors.
If you want it to behave like that then add 'degraded' to fstab. That a device is missing can have unknown reasons, the user should know better and resolve it or allow such boot. It's not automatic as there's no way to inform the user that it's degraded state.
If a device goes missing for "unknown reasons", then the machine should still work, and I'll figure out what happened when monitoring pokes me and says RAID is degraded.
It's up to you to choose at that point - is availability more important for you (add degraded to fstab), or data consistency (deal with the array first).
That's not the only purpose for it. There's three reasons I can think of that you might set up a RAID array:
* You want better uptime. (your use case)
* You want to protect from data loss. (my assumption was that this is the most common use case, but I could be wrong. This also helps with uptime because there's nothing worse for uptime than having to restore lost data from a cold backup)
* You want better performance, data integrity be damned. (RAID 0)
Booting a RAID array with a failed disk is absolutely fine idea.
How else I get access to the tools to identify the bad drive and resilver RAID on a replacement, be it in the same bay or not?
It's reasonable to want to preserve all the data you currently have—some of which probably hasn't been backed up yet—and not accept new data to be written with the durability guarantees the array was originally configured for silently violated.
Since the kernel has no way of knowing which volumes may contain important data that didn't get the chance to be backed up, it should try its best to maintain the original durability standards the filesystem was configured until some mechanism outside the kernel authorizes the relaxation of those standards.
IE (by your logic) the system should stop the writes as soon as the array became degraded.
But this is not what happens with btrfs: it would happily continue to write the data on the array until reboot.
And then suddenly it's "oh my god array is degraded!!!111 you should not write to it1111".
To add on that: I never seen for a HW RAID card to stop booting by a mere degraded state of the array. Changes in configuration of arrays, loss of more than enough for redundancy drives - yes, that would halt the boot and require the operator intervention. Array in a degraded state? Just spit the warnings to the console and boot. Nobody has the time to walk to each server with a degraded array on every reboot.
If you remove this udev rule and then add degraded mount option to fstab, it's very risky because now even a small delay in drives appearing can result in a degraded mount. And it's even possible to get a split brain situation.
Btrfs needs automatic abbreviated scrub, akin to the mdadm write intent bitmap which significantly reduces the resync operation.
Let's treat it as archived historical content it is.
That by itself is a complete deal breaker.
Anyone.
> Also if you boot with a degraded disk are you not asking for massive trouble
> not being able to boot until you add another disk
Great.
I'm just a [sys]admin who were given a task to do something on $server.
I jump around the red tape, claw out a 15 minute downtime because the task requires reboot, proceed with all that corporate dance with notification emails.
I do my thing, reboot the server and it doesn't return back online.
Suddenly I broke the server, missed maintenance window, amount of mails with CC and RE: in my mailbox grows in geometric progression and the most important - now I need to find out who were responsible for the server, contact him and [kick his ass] ask him to diagnose what is going on.
Bonus points if:
server doesn't have a meaningful BMC/iLO/iDRAC with KVM console
it does have it but it's broken for some reason eg requires Java 6 on Vista
server was configured 10 years ago by a greybeard who is not only retired but already died from the old age
server is 6000km away from any place with a replacement disks and the earliest time when you can send the replacement would be the next spring when the ice would break and taw enough for the ships to move. Of course you can hire a helo to deliver it, which get your ass chewed for an unplanned $50k expenses
This is all the things what I encountered in my admin days.
Including BL670 with incorrectly connected drives, so despite everything saying (and indicating) what the failed drive was in bay 2 it was actually in bay 1.
> then ridiculing btrfs
I ridicule btrfs for it's RAID mode not being a RAID mode by default.
RAID is about availability of data.
If you so hell bent on the data safety then btfrs should kernel panic as soon as one drive degrades. THAT would make sure someone would come and investigate what happened and no data loss.
Right?
> Right?
Panicking an already-running kernel would only serve to prevent userspace from handling the failure through mechanisms that are inherently beyond the scope and capabilities of the kernel alone (ie. stuff like alerting a sysadmin, activating a hot spare, or initiating a rebalance to restore redundancy among the remaining drives).
Perhaps the kernel should default to freezing a non-root filesystem when it becomes degraded, absent an explicit configuration permitting otherwise. But for the root filesystem, that would be counterproductive and prevent the failure from being handled gracefully.
Obviously, the tradeoffs are different for a system that is still trying to boot as opposed to one that is fully up and running.
It's not a backup. It doesn't protect against mistakes, bugs, or the server failing in some other way. What it's good for is for ensuring that work keeps happening if a disk fails. And disk failures happen to be fairly frequent, since spinning rust is a rather delicate technology.
If somebody has to connect to the machine and fiddle with it by hand it means that the system has been down, possibly for hours, when that was the exact thing you were trying to prevent by setting up RAID on it.
> What it's good for is for ensuring that work keeps happening if a disk fails.
Different services have different priorities. I want my work servers to keep running, but my home server under the desk to stop until I replace the failed drive. Btrfs gives you a choice.
It's an option to enable. If you needing to configure something is a dealbreaker then having RAID is probably not for you anyway.
For enterprise, if you dont have monitoring setup it's on you
These days, with UEFI prevalent, a system won't boot without EFI system partitions replicated (and possibly sync'd) on every drive.
It's like people justifying MySQL saying that it's okay for it to lose or corrupt data, and that transactional integrity doesn't matter as much as people say it does.
Yeah, maybe if you're website is a blog. But anyone storing real data should run away screaming from systems like this.
Maybe MySQL is sort-of-okay now? I dunno. It's possible BTRFS RAID 5 won't eat your data or crash your server regularly now.
I'm going to stay away from both anyway.
The Linux implementation is pretty awful, with it demanding that it uses the absolute `/dev/sdx` reference even if you try to build it using serials or other unchaging reference.
Replace a disk and then after the next restart it fails to bring up the RAID because `sdf` points to a different disk. At least BTRFS uses internal UUIDs so can bring up the arrays when the disk ids change.
Fairly fundamental stuff, rather than complaining about an option not being enabled by default.
Yeah, putting / on zfs was not smart. Nowadays, when the distro doesn't support it out of the box and doesn't come with kernel+bootloader+zfs tested together (i.e. proxmox, ubuntu), I wouldn't do it.
I assembled the zfs array using by-id, but after a reboot zfs reports the drives as being on /dev/sdx.
This is one of my systems using both identifications:
root@multivac[~]# zpool status
pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:01:33 with 0 errors on Fri Dec 9 03:46:34 2022
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
sdg3 ONLINE 0 0 0
errors: No known data errors
pool: multivac-slow
state: ONLINE
scan: scrub repaired 0B in 07:05:51 with 0 errors on Sun Nov 20 09:05:55 2022
config:
NAME STATE READ WRITE CKSUM
multivac-slow ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ad3395b6-ac43-453c-9ba2-cf65542cb710 ONLINE 0 0 0
2c423fe3-07a4-44cb-bc39-8b267c79d8c3 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
8ca9d5a2-04cb-47a0-987a-8b3d27326975 ONLINE 0 0 0
bec4c67f-fa41-4ab2-907f-0e5b1e052300 ONLINE 0 0 0
cache
bfef4d7f-c484-4c37-a0b7-81d59fa1e189 ONLINE 0 0 0
root@multivac[~]# blkid|grep zfs_member
/dev/sda2: LABEL="multivac-slow" UUID="4053361656876756561" UUID_SUB="363024978934966364" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="2c423fe3-07a4-44cb-bc39-8b267c79d8c3"
/dev/sdd2: LABEL="multivac-slow" UUID="4053361656876756561" UUID_SUB="10877444988141002268" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="8ca9d5a2-04cb-47a0-987a-8b3d27326975"
/dev/sdb2: LABEL="multivac-slow" UUID="4053361656876756561" UUID_SUB="9244362969528724588" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="ad3395b6-ac43-453c-9ba2-cf65542cb710"
/dev/sdc2: LABEL="multivac-slow" UUID="4053361656876756561" UUID_SUB="11658072268678819533" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="bec4c67f-fa41-4ab2-907f-0e5b1e052300"
/dev/sdg3: LABEL="boot-pool" UUID="13291833732043257716" UUID_SUB="15424416082895860251" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="b6c0b302-7599-4862-888b-be6f7a85b970"
It has gems such as:
* It won't boot on a degraded array by default, requiring manual action to mount it
* It won't complain if one of the disks is stale
* It won't resilver automatically if a disk is re-added to the array
I think the first is the killer. RAID is a High Availability measure. Your system is not Available if it fails to boot.